Benito Serna

Fix n+1 queries on rails

2023-06-22T00:01:00Z

Fix n+1 queries on Rails

Learn different ways to solve n+1 queries beyond "includes", to be confident that you can handle tasks on complex data models and fix n+1 queries if they appear.

By Benito Serna

Making things work isn't enough for you any more?

Do you feel or maybe your boss have told you directly that now you need to consider performance and scalability?

Do you feel unqualified to tackle complex tasks?

Do you still have troubles fixing some n+1 queries?

Do you have problems trying to find why active record is ignoring your "includes"?

Maybe you know the basics...

Maybe you know how to fix some n+1 queries and use "preload" or "includes" here and there, but...

Do you worry when they ask you for more?

"Can you preload just an scope?"

"Can you change the implementation to preload via has many through?"

"Shoud we use a counter cache or just preload the counts?"

"Can you persist the calculation?... just please be aware of race conditions!"

"That fix to the n+1 queries, will hurt the performance... we need to try something else"

But, what if you could...

Ship code that fix user problems taking performance in consideration.

Be confident that you can handle tasks on complex data models.

Fix n+1 queries in the best way possible taking into account different angles.

Be one of those who help the team... next week

Learn how to fix n+1 queries on rails

Start your road to fix n+1 queries like a pro, with my ebook Fix n+1 queries on rails that will help you:

Forget about… “Why active record is ignoring my includes?”
Learn different ways to solve n+1 queries beyond includes
Learn different preloading techniques to avoid n+1 queries
Learn how to count records without n+1 queries
Learn to use counter caches
Learn how to cache custom counts and other computed values with concurrency in mind

Buy the ebook

Here is the full table of contents of the book.

No more… “Why active record is ignoring my includes?”

Explain what is an n+1 queries problem
Identify when ActiveRecord will execute a query
Solve the latest comment example
Detect n+1 queries by watching the logs
Tools to detect n+1 queries

Use preloading to avoid n+1 queries

Joins does not preload
Preload direct associations
Preload nested associations
Difference between preload, eager load or includes
I prefer to use preload by default
Preload scopes with scoped associations
Simplify preloading with has many through associations
Use a “preload object” for complex preloading
Sometimes preloading can hurt the performance

Count without n+1 queries

What is the difference between count, size and length
Common mistakes with counts
How to preload counts in a list

Save counts with counter caches

Add counter cache to a new association
Custom counter cache columns names
Reset the counters for and existent association with just a few records
Reset the counters with SQL, for associations with many records
Using counter caches in your views

Caching custom counts and other computed values

Fix n+1 queries by caching computed values
Be aware that you can save incorrect values due to race conditions
Tips to avoid race conditions saving computed values

Buy the ebook

"Please help me with my n+1 problem..."

2021-05-14T00:28:00Z

You are feelling bad for asking it again… “Why active record is ignoring my includes?”.

“Is my fix right?”

Finding “a fix” can make you feel like pro, but… Does it really solves the problem? Is it elegant enough?

You know that includes can sometimes fix the problem… You are just not sure why sometimes it doesn’t.

“I know how to fix the problem!”

But what if you could? What if you knew exactly how to solve your n+1 queries problems? What if you could be the one that help others in your team?… Next week.

It’s true n+1 queries can be a pain, and they will appear… but they don’t need to appear on production.

Undestand the basics to avoid n+1 queries

Start your road to avoid n+1 queries like a pro, with my free ebook “N+1 queries basics”…

Understand preload, eager_load, includes and joins and the difference between them.
Learn to identify when active record will execute your query.
Learn the tools that can help you detect n+1 queries before they hit production.

Start solving n+1 queries problems with total confidence!

You’ll learn just enough fundamentals to be fluent preloading associations with ActiveRecord, and start helping your team to avoid n+1 queries on production.

Start shipping efficient code, sign up and download my free ebook and you’ll be enjoying the power of Rails again, in no time.

MdRecord: File-backed models for Rails

2025-12-29T12:00:00Z

I just released md_record, a Ruby gem that provides an ActiveRecord-like interface for markdown files with YAML frontmatter.

Why?

I needed a simple way to manage content like blog posts and guides in my Rails apps without a database. I wanted something that felt like ActiveRecord but worked with markdown files.

Installation

gem "md_record"

Basic Usage

Create a model that inherits from MdRecord::Base:

class Post < MdRecord::Base
end

By default, files are loaded from app/views/posts/ (based on the model name).

Create markdown files with YAML frontmatter:

---
title: My First Post
description: A short description
published: true
published_at: 2024-12-29
position: 1
---

# Content here

Your markdown content...

Then use it like ActiveRecord:

Post.all        # All posts
Post.published  # Only published posts
Post.find(slug) # Find by slug or raise MdRecord::RecordNotFound

Custom Path

class Article < MdRecord::Base
  md_record_path "content/articles"
end

Categorized Content

For content organized in categories (like documentation or guides):

class Guide < MdRecord::Base
  include MdRecord::Categorizable

  md_record_categories(
    "getting-started" => { name: "Getting Started", position: 1 },
    "advanced" => { name: "Advanced", position: 2 }
  )
end

Files are organized in subfolders:

app/views/guides/
  getting-started/
    first-steps.md
  advanced/
    custom-config.md

Then you can use:

Guide.categories  # Returns array of MdRecord::Category objects
guide.category?("getting-started")  # Check if guide belongs to category
guide.category_name  # "Getting Started"

Rails Integration

The gem includes a Railtie that automatically maps MdRecord::RecordNotFound to a 404 response, just like ActiveRecord::RecordNotFound.

Post.find("non-existent") # raises MdRecord::RecordNotFound
# Rails returns 404

Rendering to HTML

Each record can render its markdown content to HTML:

post = Post.find("my-post")
post.to_html  # Returns HTML string

Fixing a bug in my lateral joins queries with rails

2025-01-21T12:00:00Z

I have some posts about using lateral joins in rails to fetch the “top N” of each record. Some months ago Ben Sheldon helped me see a performance problem on the queries that I was using.

Here I try to explain the problem, the solution proposed by him, and also an introduction about a tool he published to build this kind of “top N” queries using lateral joins.

The problem

In his words, “The outermost condition of the association (e.g. WHERE user_id IN ($1, ...)) doesn’t get pushed down in the current query. This leads to the lateral join being applied to all records before the condition is applied.”

The EXPLAIN helps to see this problem. This is the example that has the problem:

class User < ActiveRecord::Base
  has_many :posts
  has_many :last_posts, -> { last_n_per_user(3) }, class_name: "Post"
end

class Post < ActiveRecord::Base
  scope :last_n_per_user, ->(n) {
    sql = <<-SQL
      JOIN LATERAL (
        SELECT * FROM posts
        WHERE user_id = users.id
        ORDER BY id DESC LIMIT :limit
      ) AS selected_posts ON TRUE
    SQL

    selected_posts = User
    .select("selected_posts.*")
    .joins(User.sanitize_sql([sql, limit: n]))

    from(selected_posts, "posts")
  }
end

puts "users count: 100"
puts "posts count: 10000"
users = User.preload(:last_posts).limit(2)
puts users.explain

This returns…

users count: 100
posts count: 10000
EXPLAIN SELECT "posts".* FROM (SELECT selected_posts.* FROM "users" JOIN LATERAL (
          SELECT * FROM posts
          WHERE user_id = users.id
          ORDER BY id DESC LIMIT 3
        ) AS selected_posts ON TRUE) posts WHERE "posts"."user_id" IN ($1, $2) [["user_id", 1], ["user_id", 2]]
                                               QUERY PLAN
--------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.29..1139.25 rows=100 width=25)
   ->  Seq Scan on users  (cost=0.00..2.00 rows=100 width=8)
   ->  Subquery Scan on selected_posts  (cost=0.29..11.36 rows=1 width=25)
         Filter: (selected_posts.user_id = ANY ('{1,2}'::integer[]))
         ->  Limit  (cost=0.29..11.32 rows=3 width=25)
               ->  Index Scan Backward using posts_pkey on posts  (cost=0.29..368.29 rows=100 width=25)
                     Filter: (user_id = users.id)

In this example I have 100 users in the database, and even when I used limit(2) in the query, the plan shows 100 rows, instead of 2.

Proposed solution

The proposed solution by Ben Sheldon, looks like this…

class User < ActiveRecord::Base
  has_many :posts
  has_many :last_posts, -> { last_n_per_user(3) }, class_name: "Post"
end

class Post < ActiveRecord::Base
  scope :last_n_per_user, ->(n) {
    selected_posts_table = Arel::Table.new('selected_posts')

    sql = <<-SQL
      JOIN LATERAL (
        SELECT * FROM posts
        WHERE user_id = users.id
        ORDER BY id DESC LIMIT :limit
      ) AS #{selected_posts_table.name} ON TRUE
    SQL

    selected_posts = User
      .select(User.arel_table["id"].as("user_id"))
      .select(Post.column_names.excluding("user_id").map { |column| selected_posts_table[column] })
      .joins(User.sanitize_sql([sql, limit: n]))

    from(selected_posts, "posts")
  }
end

ActiveRecord::Base.connection.execute("ANALYZE users, posts")
puts "users count: #{User.count}"
puts "posts count: #{Post.count}"
users = User.preload(:last_posts).limit(2)
puts users.explain

Here using Arel, he SELECT users.id as user_id and then enumerates the columns from the Post record excluding the user_id.

In his words, he does it because… “The error happens because the query in the sub-select has two user_id columns: one from users.id AS user_id and then the user_id from latest.*. That’s why it’s necessary to enumerate the columns to omit the latter. It’s not great looking 😞”

And now the EXPLAIN shows that the scan for users is limited to 2 users.

users count: 100
posts count: 10000
EXPLAIN SELECT "posts".* FROM (SELECT "users"."id" AS user_id, "selected_posts"."id", "selected_posts"."title" FROM "users" JOIN LATERAL (
          SELECT * FROM posts
          WHERE user_id = users.id
          ORDER BY id DESC LIMIT 3
        ) AS selected_posts ON TRUE) posts WHERE "posts"."user_id" IN ($1, $2) [["user_id", 1], ["user_id", 2]]
                                            QUERY PLAN
--------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.29..25.02 rows=6 width=29)
   ->  Seq Scan on users  (cost=0.00..2.25 rows=2 width=8)
         Filter: (id = ANY ('{1,2}'::bigint[]))
   ->  Limit  (cost=0.29..11.32 rows=3 width=25)
         ->  Index Scan Backward using posts_pkey on posts  (cost=0.29..368.29 rows=100 width=25)
               Filter: (user_id = users.id)
(6 rows)

A new tool to define top-N-per-group associations using lateral joins

Ben Sheldon published a new tool called activerecord-has_some_of_many that will give you methods like has_one_of_many and has_some_of_many to automatically build associations like the one in this example.

It will help you build “top N” queries using lateral joins that you will be able to preload (or includes) to avoid N+1 queries and also are compatible with typical queries and batch methods (find_each, in_batches, find_in_batches).

Simple Searchable module for searching with Rails and SQLite's LIKE

2024-02-09T23:33:00Z

Yesterday, I found myself struggling with “litesearch”, and in my frustration, I decided to create a simple “Searchable” module to perform a “LIKE” search but with a cleaner interface/API.

If you’re looking for a straightforward way to implement a search feature in your application using “LIKE”, this post might be helpful to you.

…However, it turns out the issue wasn’t with “litesearch” at all. It was actually related to the parallelization of the specs. For now, I’ll continue using “litesearch” instead of the code provided in this post. Nevertheless, I wanted to share it because it might be useful to you (or even to myself in the future).

The Searchable Module

You can include the following code in app/models/concerns/searchable.rb:

module Searchable
  extend ActiveSupport::Concern

  included do
    cattr_accessor :searchable_fields, :searchable_associations_fields
  end

  class_methods do
    def search_on(*searchable_fields, **searchable_associations_fields)
      self.searchable_fields = searchable_fields
      self.searchable_associations_fields = searchable_associations_fields

      scope :search, ->(query) do
        left_outer_joins(searchable_joins).where(searchable_where_clause, query: "%#{query}%").distinct
      end

      scope :maybe_search, ->(query) do
        search(query) if query.present?
      end
    end

    def searchable_joins
      searchable_associations_fields.keys
    end

    def searchable_where_clause
      all_normalized_searchable_fields.map { |field| "#{field} LIKE :query" }.join(" OR ")
    end

    def all_normalized_searchable_fields
      normalized_searchable_fields + normalized_searchable_associations_fields
    end

    def normalized_searchable_fields
      searchable_fields.map { |field| [table_name, field].join(".") }
    end

    def normalized_searchable_associations_fields
      searchable_associations_fields.flat_map do |association, fields|
        table_name = reflect_on_association(association).klass.table_name
        Array.wrap(fields).map { |field| [table_name, field].join(".") }
      end
    end
  end
end

How to Use It

You can use the module in your models like this:

class Contact < ApplicationRecord
  include Searchable

  search_on :name, :last_name, :full_name, :email, :current_company, :phone
end

You can also search on associations using a hash key with the name of the association and a list of attributes to search on.

For example:

class QuoteRequest < ApplicationRecord
  include Searchable
  belongs_to :requester, class_name: "Contact"

  search_on :custom_id, :brand_name, :product_name, :more_info, :quantity, requester: :full_name
end

In this example, you’ll also search on the requester.full_name attribute, which corresponds to the "contacts.full_name" column.

If you want to search on multiple fields of the “requester”, you can do something like this:

class QuoteRequest < ApplicationRecord
  include Searchable
  belongs_to :requester, class_name: "Contact"

  search_on :custom_id, :brand_name, :product_name, :more_info, :quantity, requester: [:email, :full_name]
end

The tests to show how each part works

I didn’t write a lot of tests for it, and I didn’t write them in a generic way. However, I will show them to you because they can help you understand the purpose of each part of the code.

require "test_helper"

class SearchableTest < ActiveSupport::TestCase
  test "quote_requests metadata" do
    # Keeps the list of fields to search on the record
    assert_equal QuoteRequest.searchable_fields, [:custom_id, :brand_name, :product_name, :more_info, :quantity]

    # Keeps a hash with the list of fields to search for each association
    assert_equal QuoteRequest.searchable_associations_fields, {requester: :full_name}

    # Builds the list of fields with the table_name, to be used later in the query
    assert_equal QuoteRequest.normalized_searchable_fields, ["quote_requests.custom_id", "quote_requests.brand_name", "quote_requests.product_name", "quote_requests.more_info", "quote_requests.quantity"]

    # Builds the list of fields from the associations with the table_name, to be used later in the query
    assert_equal QuoteRequest.normalized_searchable_associations_fields, ["contacts.full_name"]

    # The string to be used in the where clause
    assert_equal QuoteRequest.searchable_where_clause, "quote_requests.custom_id LIKE :query OR quote_requests.brand_name LIKE :query OR quote_requests.product_name LIKE :query OR quote_requests.more_info LIKE :query OR quote_requests.quantity LIKE :query OR contacts.full_name LIKE :query"
  end

  test "contacts example" do
    result = Contact.search("Pedro")
    assert_equal result.count, 1
    assert_equal result.first, contacts(:one)

    result = Contact.search("Perez")
    assert_equal result.count, 1
    assert_equal result.first, contacts(:two)
  end

  test "quote_requests example" do
    result = QuoteRequest.search("Pedro")
    assert_equal 1, result.count
    assert_equal quote_requests(:one), result.first

    result = QuoteRequest.search("Perez")
    assert_equal 1, result.count
    assert_equal quote_requests(:two), result.first
  end
end

Maybe you can use it

While I won’t be using this module for now due to my resolution of the issue with “litesearch”, perhaps you may find it useful.

Whether you’re interested in adapting it for PostgreSQL or exploring alternative search solutions, I hope this code may serve as a helpful reference or source of inspiration.

A "read-more" behavior truncating by number of lines with css line-clamp and Stimulus.js

2021-04-03T00:28:00Z

I have already shared a way of implementing a “read-more” behavior truncating by the number of lines instead of the number of words.

But now I want to share how you can do it using the line-clamp css property.

The example

Visit example page

The html and css

Your are going to need…

A linesValue to configure the number of lines that we want to display,
A class to hide the buttons and a class to truncate the content…
3 targets, the content, the moreButton and the lessButton…
3 actions
- resize@window->read-more#render - to calculate the truncation on rezise.
- click->read-more#showMore
- click->read-more#showLess

html { line-height: 1.5 }
.hide { display: none; }
.line-clamp {
  overflow : hidden;
  text-overflow: ellipsis;
  display: -webkit-box;
  -webkit-box-orient: vertical;
}

<div data-controller="read-more"
     data-read-more-lines-value="3"
     data-read-more-hide-class="hide"
     data-read-more-truncate-class="line-clamp"
     data-action="resize@window->read-more#render">
  <p data-read-more-target="content">
    The content that you want to truncate...
  </p>
  <button class="hide"
    data-read-more-target="moreButton"
    data-action="read-more#showMore">
    Show more
  </button>
  <button class="hide"
    data-read-more-target="lessButton"
    data-action="read-more#showLess">
    Show less
  </button>
</div>

The javascript

You are going to use line-clamp to truncate the content, assigning the configured linesValue to the property -webkit-line-clamp.

But you are going to do it just if the height is greater that the expectedHeigt that is the product of the linesValue with the lineHeight.

import { Controller } from "@hotwired/stimulus";

export default class extends Controller {
  static targets = ["content", "moreButton", "lessButton"];
  static classes = ["truncate", "hide"]
  static values = { lines: Number }

  connect() {
    this.render()
  }

  render() {
    this.showAllContent()

    if (this.height() > this.expectedHeight()) {
      this.showLess()
    } else {
      this.showAllContent()
      this.hide(this.moreButtonTarget);
      this.hide(this.lessButtonTarget);
    }
  }

  showMore() {
    this.showAllContent();
    this.hide(this.moreButtonTarget);
    this.show(this.lessButtonTarget);
  }

  showLess() {
    this.truncateContent();
    this.hide(this.lessButtonTarget);
    this.show(this.moreButtonTarget);
  }

  showAllContent() {
    this.contentTarget.classList.remove(this.truncateClass);
  }

  truncateContent() {
    this.contentTarget.style["-webkit-line-clamp"] = this.linesValue;
    this.contentTarget.classList.add(this.truncateClass);
  }

  show(target) {
    target.classList.remove(this.hideClass)
  }

  hide(target) {
    target.classList.add(this.hideClass)
  }

  lineHeight() {
    let style = window.getComputedStyle(this.contentTarget)
    return parseFloat(style.lineHeight, 10);
  }

  height() {
    return this.contentTarget.offsetHeight;
  }

  expectedHeight() {
    return this.linesValue * this.lineHeight();
  }
}

Other ways to do it…

If for some reason, you can’t use line-clamp, maybe you can try doing the truncation with custom javascript.

And if you can truncate by number of characters, you can check this example using the “truncate” helper.

Copy shoes.rb and use stacks and flows to build layouts

2024-01-24T00:07:00Z

I was reading about shoes.rb and, while scanning the book and walkthrough, I discovered a pattern that caught my attention for its power and simplicity.

The combination of “stacks” and “flows” allows you to build complex layouts, like the following example in Ruby:

Shoes.app do
  background "#EFC"
  border("#BE8", strokewidth: 6)

  stack(margin: 12) do
    para "Enter your name"
    flow do
      edit_line
      button "OK"
    end
  end
end

What are a stacks and a flows?

A stack places one thing on top of another, similar to blocks in HTML.
A flow arranges items next to each other on a horizontal line, like inline blocks in HTML.

With CSS, you can implement them like this:

.stack {
  display: flex;
  gap: 1em;
  flex-direction: column;
  justify-content: flex-start;
}

.flow {
  display: flex;
  gap: 0.5em;
  align-items: baseline;
  flex-wrap: wrap;
}

Here’s a pen with some examples from the walkthrough to help you visualize how you can combine these two CSS rules:

View CodePen Example

See the Pen Untitled by Benito Serna (@bhserna) on CodePen.

Personal testing of the idea

I have been using a slightly modified version of these CSS classes in a personal project where I am also using open-props.

These are the CSS classes that I am using:

.stack {
  display: flex;
  gap: var(--size-3);
  flex-direction: column;
  justify-content: flex-start;

  &.sm {
    gap: var(--size-1);
  }

  &.lg {
    gap: var(--size-7);
  }
}

.flow {
  display: flex;
  gap: var(--size-2);
  align-items: baseline;
  flex-wrap: wrap;

  &.centered {
    align-items: center;
  }

  &.justify-between {
    justify-content: space-between;
  }

  &.sm {
    gap: var(--size-1);
  }

  &.lg {
    gap: var(--size-3);
  }
}

I don’t have enough experience with them to be sure that you should adopt these classes in all your projects, but I think you should at least test them in your personal projects and play around with them.

If you use something like Tailwind CSS, maybe you can write a partial or component to introduce this abstraction.

Maybe this names can break your UI kit

The names “stack” and “flow” are used as UI elements by some HTML or native UI kits.

So, if you are testing them, check if these classes are not already used in your project.

Fix n+1 queries by caching computed values

2023-07-19T23:19:00Z

N+1 queries are not always a problem, but I have seen that most of the n+1 queries that are really a problem are when we need to fetch data to compute something.

Here I will try to share some examples of posible expensive computations candidates to be cached and some patterns that you could use to save different kind of values.

Some examples of possible expensive computations

I think the most common computation in many apps will be a count. It is that common that rails already have “counter caches”.

But sometimes you will need to save counts where a counter cache won’t be enough, like:

Counts for “has many through” associations.
Counts for a scope of the association, like just the “positive reactions”, or the “completed orders”.

Or maybe other examples where you need other type of aggregation like:

The balance in an account
The a profile completeness percentage
A TIR, current value of a portafolio, or other financial calculations

Should you always cache computations?

No, not really, you should evaluate the pros and cons in each situation.

Sometimes you will need a way to cache the value since the first implementation, but other times it will be better to wait and learn a little more about the problem and cache the value until is really necessary.

Sometimes pre-compute the value will be very easy, but sometimes could be more expensive or very hard, or maybe just not needed.

Patterns to save computed values

There are many ways to save a value, but here I will show you three patterns that I use a lot.

Using a before_save callback

You could use it for calculations that involve information on the same record, like a profile completeness calculation.

class Profile < ApplicationRecord
  before_save :set_completeness_percentage

  def set_completeness_percentage
    self.completeness_percentage = calculate_completeness_percentage
  end
end

When you use a before_save callback, you don’t need to save the record, you just need to change the value and the record will be saved including the fields you modified.

Using after_touch callback

You could use it when you need to save or change a value for things that happen in other records, like updating the balance when an entry is created or updated:

class Account < ApplicationRecord
  has_many :entries

  after_touch :update_balance

  def update_balance
    update(balance: entries.sum(:amount))
  end
end

class Entry < ApplicationRecord
  belongs_to :account, touch: true
end

Using after_commit + background job

Use it for calculations are expensive to run on the same process and is better to run them asynchronously.

class Profile < ApplicationRecord
  after_commit :set_completeness_percentage_later

  def set_completeness_percentage_later
    Profile::SetCompletenessPercentageProfileJob.perform_later(self)
  end

  def set_completeness_percentage
    self.completeness_percentage = calculate_completeness_percentage
  end
end

class Profile::SetCompletenessPercentageProfileJob < ApplicationJob
  def perform(profile)
    profile.set_completeness_percentage
    profile.save!
  end
end

Be aware of race conditions

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result thanks to race conditions.

I have other article that can help you visualize how race conditions can make you save incorrect values even when your calculation is correct.

You can read it on: Saving incorrect computed values thanks to race conditions

Tools to help you detect n+1 queries

2021-03-01T12:57:00Z

There are many tools that can help you detect n+1 queries in different ways.

This is a little reference of some of those tools:

Strict loading
Rack mini profiler
Bullet
Prosopite
n plus one control

You don’t need to use all of them, but is good to know that they exists and how they can help you.

Rails strict_loading

How it can help you?

You can add #strict_loading! to any record to prevent lazy loading of associations. Strict loading will cascade down from the parent record to all the associations to help you catch any places where you may want to use preload instead of lazy loading.

On a record:

user = User.first
user.strict_loading!
user.comments.to_a
=> ActiveRecord::StrictLoadingViolationError

On a relation:

user = User.strict_loading.first
user.comments.to_a
=> ActiveRecord::StrictLoadingViolationError

On an association definition:

class User < ApplicationRecord
  has_many :comments, strict_loading: true
end

class Comment < ApplicationRecord
  belongs_to :user
end

user = User.first
user.comments.to_a
=> ActiveRecord::StrictLoadingViolationError

Per model configuration:

class User < ApplicationRecord
  self.strict_loading_by_default = true

  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :user
end

user = User.first
user.comments.to_a
=> ActiveRecord::StrictLoadingViolationError

If you want to enable strict loading by default you can do it with:

config.active_record.strict_loading_by_default = true

If you can change the action on strict loading violation from :raise (default) to :log you can do it with:

config.active_record.action_on_strict_loading_violation = :log

References

Rack mini profiler

How it can help you?

It can help you with more than just detecting n+1 queries. It is a production and development profiler, it allows you to quickly isolate performance bottlenecks, both on the server and client.

It can help you with:

Database profiling
Call-stack profiling
Memory profiling

How does the report looks like?

It displays a speed badge for every html page that if you click it, it will show you a page with the profiling information of the current page.

And if you click on of the sql queries count, it will show you a list with all the queries…

Although it will not tell you exactly that you have an n+1 quieries problem, it can help you a lot to visualize it.

References

Bullet

How can it help you?

It will watch your queries while you develop your application and notify you when it detects one of this problems:

n+1 queries
eager-loaded associations which are not used
unnecessary COUNT queries which could be avoided with a counter cache

You can use it on the development and testing environments.

Sometimes Bullet may notify you of query problems you don’t care to fix, or which come from outside your code. You can whitelist these to ignore them.

How does the error report looks like?

For n+1 queries…

GET /posts
USE eager loading detected
  Post => [:comments]
  Add to your query: .includes([:comments])
Call stack
  /Users/benitoserna/code/bullet-test/app/views/posts/index.html.erb:20:in `map'
  /Users/benitoserna/code/bullet-test/app/views/posts/index.html.erb:20:in `block in _app_views_posts_index_html_erb__1178069968615334744_70147771830640'
  /Users/benitoserna/code/bullet-test/app/views/posts/index.html.erb:16:in `_app_views_posts_index_html_erb__1178069968615334744_70147771830640'

For unused eager loading…

GET /posts
AVOID eager loading detected
  Post => [:comments]
  Remove from your query: .includes([:comments])
Call stack

References

github/flyerhzm/bullet

Prosopite

How can it help you?

Prosopite is able to auto-detect Rails N+1 queries, but it also can help you detect some cases where bullet will give you false positives or false negatives.

Prosopite monitors all SQL queries using the Active Support instrumentation and looks for a pattern which is present in all N+1 query cases: More than one queries have the same call stack and the same query fingerprint.

You can use it on the development and testing environments.

Compared to bullet, Prosopite can auto-detect the following extra cases of N+1 queries:

N+1 queries after record creations (usually in tests)

FactoryBot.create_list(:leg, 10)

Leg.last(10).each do |l|
  l.chair
end

Not triggered by ActiveRecord associations

Leg.last(4).each do |l|
  Chair.find(l.chair_id)
end

First/last/pluck of collection associations

Chair.last(20).each do |c|
  c.legs.first
  c.legs.last
  c.legs.pluck(:id)
end

Changing the ActiveRecord class with #becomes

Chair.last(20).map{ |c| c.becomes(ArmChair) }.each do |ac|
  ac.legs.map(&:id)
end

Mongoid models calling ActiveRecord

class Leg::Design
  include Mongoid::Document
  ...
  field :cid, as: :chair_id, type: Integer
  ...
  def chair
    @chair ||= Chair.where(id: chair_id).first!
  end
end

Leg::Design.last(20) do |l|
  l.chair
end

How does the error report looks like?

The report will show you the N+1 queries detected and the call stack.

N+1 queries detected:
  SELECT `users`.* FROM `users` WHERE `users`.`id` = 20 LIMIT 1
  SELECT `users`.* FROM `users` WHERE `users`.`id` = 21 LIMIT 1
  SELECT `users`.* FROM `users` WHERE `users`.`id` = 22 LIMIT 1
  SELECT `users`.* FROM `users` WHERE `users`.`id` = 23 LIMIT 1
  SELECT `users`.* FROM `users` WHERE `users`.`id` = 24 LIMIT 1
Call stack:
  app/controllers/thank_you_controller.rb:4:in `block in index'
  app/controllers/thank_you_controller.rb:3:in `each'
  app/controllers/thank_you_controller.rb:3:in `index':
  app/controllers/application_controller.rb:8:in `block in <class:ApplicationController>'

References

github/charkost/prosopite

NPlusOneControl

How can it help you?

It gives you rspec and minitest matchers to prevent the n+1 queries problem.

It evaluates the code under consideration several times with different scale factors to make sure that the number of DB queries behaves as expected (i.e. O(1) instead of O(N)).

So, it’s for performance testing and not feature testing.

How does the error report looks like?

In the default mode it can give you something like this.

Expected to make the same number of queries, but got:
  10 for N=2
  11 for N=3
Unmatched query numbers by tables:
  resources (SELECT): 2 != 3
  permissions (SELECT): 4 != 6

And in the “verbose” mode, it can give you something like this…

Expected to make the same number of queries, but got:
  2 for N=2
  3 for N=3
Unmatched query numbers by tables:
  resources (SELECT): 2 != 3
Queries for N=2
   SELECT "resources".* FROM "resources" WHERE "resources"."deleted_at" IS NULL
   ↳ app/controllers/resources_controller.rb:32:in `index'
   ...
Queries for N=3
   ...

References

Truncate in the middle with truncate rails helper

2023-06-19T23:08:00Z

Imagine that you want to truncate a filename, but you want to keep showing the extension of the file. Like “A big file name that…awesome.pdf”. How would you do it?

The omission parameter

With the :omission string the last characters will be replaced (defaults to “…”) for a total length not exceeding length:

string = "And they found that many people were sleeping better."
string.truncate(25, omission: '... (continued)')
# => "And they f... (continued)"

So, you can pass as the omssion string, the last characters of your string like this:

filename = "And they found that many people were sleeping better.pdf"
filename.truncate(25, omission: "... #{filename.last(10)}")
# => "And they fo... better.pdf"

A form with two buttons with formation and formmethod

2023-06-08T12:17:00Z

Imagine that you are building a custom CMS. Within the form to edit an Article, you need to have two buttons: a normal “Save” button and a new “Save and publish” button. And maybe, additionally, you will need a third button to delete the article.

To achieve this, you can use the formaction and formmethod attributes.

Changing the action with `formaction`

To add the “Save and publish” button, use the formaction attribute to override the action in the form:

<%= form_with model: article do |form| %>
  ...
  <%= form.button "Save" %>
  <%= form.button "Save and publish", formaction: save_and_publish_article_path %>
<% end %>

This sends the form, along with all the field information, to the path you defined in formaction. To process the information in a new controller action, define a route:

resources :articles do
  patch :save_and_publish, on: :member
end

Then, create the new controller action:

def save_and_publish
  if @article.update_and_publish(article_params)
    redirect_to article
  else
    render :edit, status: :unprocessable_entity
  end
end

Changing the method with `formmethod`

To add the “Delete” button, use the formmethod attribute to let HTML override the declared method attribute:

<%= form_with model: article do |form| %>
  ...
  <%= form.button "Save" %>
  <%= form.button "Delete", formmethod: :delete, data: { confirm: "Are you sure?" } %>
<% end %>

Most browsers don’t support overriding form methods declared through formmethod other than “GET” and “POST”. Rails works around this issue by emulating other methods over POST through a combination of formmethod, value, and name attributes.

<form accept-charset="UTF-8" action="/articles/1" method="post">
  <input name="_method" type="hidden" value="patch" />
  <input name="authenticity_token" type="hidden" value="..." />
  <!-- ... -->

  <button type="submit" name="button">Save</button>
  <button type="submit" formmethod="post" name="_method" value="delete" data-confirm="Are you sure?">Delete</button>
</form>

What to do when you need a button_to within a form in Rails

2023-06-05T12:12:00Z

Imagine that you have a form to update a record (let’s say a product record) and inside the form, you are showing a list of images, and each image needs a button to remove it. You tried to use button_to but it doesn’t work because in html you can have form within a form. What do you do?

You can use the “form” attribute

You can use a button tag and define its form attribute. The value should be the id of a form located anywhere in the document.

For example, in the next code, for each persited image, we call render a button with a form attribute “delete_image_#{image.id}”.

<%= form_with(model: product) do |form| %>
  ...
  <div class="product-images">
    <% product.persisted_images.each do |image| %>
      <figure>
        <%= image_tag image %>
        <%= tag.button(
          "Remove",
          type: "submit",
          form: "delete_image_#{image.id}") %>
      </figure>
    <% end %>
  </div>
  ...
<% end %>

And later in the page, for each persisted image, we can render a hidden form with id “deleteimage#{image.id}” that will match the previous button.

<% product.persisted_images.each do |image| %>
  <%= form_with(
    model: image,
    url: product_image_path(product, image),
    method: :delete,
    id: "delete_image_#{image.id}") do %>
  <% end %>
<% end %>

Simple image manager with active storage

2023-05-22T23:25:00Z

If you want to add images to a record but you don’t want to use a JavaScript plugin or write any custom JavaScript, you can use a regular file field, Active Storage, and vanilla Rails.

If you want to be able to:

Attach many images on create
Keep adding many images on every update
Display the images in a gallery
Be able to remove the images one by one

Here is a tutorial to help you accomplish that.

Example: A product with many attached images

To use as an example for the tutorial, we are going to have a Product record that has many attached images like this:

class Product < ApplicationRecord
  has_many_attached :images
end

Add many images on create and every update

If you want to add many attachments to a record using just a file field, but you don’t want to remove the previous images from the record on every update, you can use a virtual attribute “new_images”.

Instead of using the :images attribute, you can add a virtual :new_images attribute, using an attr_reader and a custom writer like this:

class Product < ApplicationRecord
  attr_reader :new_images

  has_many_attached :images

  def new_images=(images)
    self.images.attach(images)
  end
end

And use that attribute in your file_field:

<%= form_with(model: product) do |form| %>
    <%= form.label :new_images %>
    <%= form.file_field :new_images, multiple: true %>
  <%= form.submit %>
<% end %>

Also, update your “params method” in the controller:

def create
  @product = @site.products.new(product_params)
  #...
end

def update
  @product.update(product_params)
  # ...
end

private

def product_params
  params.require(:product).permit(new_images: [])
end

This way, when you assign new images, your record will attach the received images instead of updating the images field with the new images.

Display the images

To display the images, you can put an HTML like this:

<div class="product-images">
  <% product.persisted_images.each do |image| %>
    <figure>
      <%= image_tag image %>
    </figure>
  <% end %>
</div>

Where “persisted_images” is:

def persisted_images
  images.select(&:persisted?)
end

Maybe you don’t need this, but I use this HTML inside the form, and I had some troubles when the record was not valid; it was showing images without content, this solved my problem.

Remove the images one by one

If you are not displaying the images inside the form, maybe you can put a button_to remove the image near the image.

<div class="product-images">
  <% product.persisted_images.each do |image| %>
    <figure>
      <%= image_tag image %>
      <%= button_to(
        "Remove",
        product_image_path(product, image),
        method: :delete) %>
    </figure>
  <% end %>
</div>

But if you want to put the images inside your form, you can’t use a button_to because you can’t have a form inside a form.

So one thing you can do is to have a button that will trigger a form that is outside the main form, like this:

<%= form_with(model: product) do |form| %>
  <div>
    <%= form.label :new_images %>
    <%= form.file_field :new_images, multiple: true %>
  </div>

  <div class="product-images">
    <% product.persisted_images.each do |image| %>
      <figure>
        <%= image_tag image %>
        <%= tag.button(
          "Remove",
          type: "submit",
          form: "delete_image_#{image.id}") %>
      </figure>
    <% end %>
  </div>

  <div>
    <%= form.submit %>
  </div>
<% end %>

<% product.persisted_images.each do |image| %>
  <%= form_with(
    model: image,
    url: product_image_path(product, image),
    method: :delete,
    id: "delete_image_#{image.id}") do %>
  <% end %>
<% end %>

In the form, we are using a button tag of type submit that will trigger a form with id "delete_image_#{image.id}".

<%= tag.button(
  "Remove",
  type: "submit",
  form: "delete_image_#{image.id}") %>

And we are creating a form for each image specifying that id:

<% product.persisted_images.each do |image| %>
  <%= form_with(
    model: image,
    url: product_image_path(product, image),
    method: :delete,
    id: "delete_image_#{image.id}") do %>
  <% end %>
<% end %>

Then on the controller that handles the product_image_path, you can remove the image with purge.

class ImagesController < ApplicationController
  before_action :set_product

  def destroy
    @product.images.find(params[:id]).purge
    #...
  end

  private

  def set_product
    @product = Product.find(params[:product_id])
  end
end

Sample Code

I have a sample application with code available at https://github.com/bhserna/simpleimagemanagment.

Add many attachments without deleting previous ones using ActiveStorage

2023-05-16T12:09:00Z

If you want to add many attachments to a record using just a file field, but you don’t want to remove the previous images from the record on every update, like in the following code:

<%= form_with(model: product) do |form| %>
    <%= form.label :images %>
    <%= form.file_field :images, multiple: true %>
  <%= form.submit %>
<% end %>

And instead, on every update you want to add the new images to the record, here is a simple workaround…

A virtual new_images attribute

Instead of using the :images attribute, you can add a virtual :new_images attribute, using an attr_reader and a custom writer like this:

class Product < ApplicationRecord
  attr_reader :new_images

  has_many_attached :images

  def new_images=(images)
    self.images.attach(images)
  end
end

And use that attribute in your file_field:

<%= form_with(model: product) do |form| %>
    <%= form.label :new_images %>
    <%= form.file_field :new_images, multiple: true %>
  <%= form.submit %>
<% end %>

Also, update your “params method” in the controller:

def create
  @product = @site.products.new(product_params)
  #...
end

def update
  @product.update(product_params)
  # ...
end

private

def product_params
  params.require(:product).permit(new_images: [])
end

This way, when you assign new images, your record will attach the received images instead of updating the images field with the new images.

Use after_touch to avoid to handle race conditions saving computed values

2023-03-02T23:16:00Z

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result thanks to race conditions.

I have already shared an exercise to help you get more sensitivity about when an implementation can save a wrong value thanks to race conditions.

Here I want to share one tip to help you avoid save the wrong value due race conditions while trying to save a computed value.

Using the account balance as an example

To talk about something concrete I will use the “account balance” as an example, but you can use this approach for different types of calculations.

The account balance example

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

Imagine each account will need to create many entries concurrently, maybe on different background jobs or different requests. So if you want to calculate the balance and save it just after an entry is created, you could have problems with race conditions.

Tip: Use after_touch to trigger the operation

As far as I understand, if you are using Postgres with the default isolation level Read Committed, you can compute and save the value on an after_touch like this:

class Account < ExampleRecord
  has_many :entries

  after_touch :update_balance

  def create_entry(amount:)
    entries.create(amount: amount)
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Why does it work?

Or at least, why do I think that it works…

Apparently after_touch works something like this:

There is a change in the association.
Within the same transaction, rails updates the associated record (account).
Then it performs the other operation that you have configured (update_balance).

PostgreSQL uses by default the Read Committed isolation level.

In this mode (as far as I understand), if within a transaction there is an UPDATE of a row and in another concurrent transaction an UPDATE of the same row is attempted, PostgreSQL will wait for the first transaction to be committed before continuing with the second one.

This makes the operations we perform on after_touch to be isolated and avoids the problem of saving a wrong error due to another transaction registering another entry for the same account concurrently.

Do you want to test it?

You can check the examples to explore possible race conditions when caching custom computed values in Rails.

And more specifically you can test this example that uses after_touch.

Are we avoiding race conditions?

We are not avoiding race conditions, other problems can happen, but we are at least avoiding saving the wrong value due to race conditions.

Is this a perfect fix?

No, if the calculation of the value that you are trying to save takes a lot of time and your application is really concurrent, it can queue up many transactions and cause other problems.

Also the reason why it works, can be a little obscure.

Do you know other problems with this solution?

If you have experience with other problems with this solution, please leave a comment =)

Use with_lock to handle race conditions saving computed values

2023-03-01T00:00:00Z

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result thanks to race conditions.

I have already shared an exercise to help you get more sensitivity about when an implementation can save a wrong value thanks to race conditions.

Here I want to share one tip to help you avoid save the wrong value due race conditions while trying to save a computed value.

Using the account balance as an example

To talk about something concrete I will use the “account balance” as an example, but you can use this approach for different types of calculations.

The account balance example

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

Tip: Use with lock in the full operation

Instead of “just” saving the balance and maybe save a wrong value, you can wrap the full operation in a with_lock block.

By calling with_lock you will start a transaction and acquire a lock for the row in one go. The block is called from within a transaction, and the object is already locked.

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    with_lock do
      # This block is called within a transaction,
      # and account is already locked
      entries.create(amount: amount)
      update_balance
    end
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

You can learn more of with_lock and the ActiveRecord::Locking::Pessimistic module on the rails api.

Are we avoiding race conditions?

We are not avoiding race conditions, other problems can happen, but we are at least avoiding saving the wrong value due to race conditions.

Is this a perfect fix?

No, locks can produce other type of problems, like deadlocks.

Do you know other problems with this solution?

If you have experience with other problems with this solution, please leave a comment =)

Pick a safe previous date to avoid race conditions saving computed values

2023-02-14T00:39:00Z

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result thanks to race conditions.

I have already shared an exercise to help you get more sensitivity about when an implementation can save a wrong value thanks to race conditions.

Here I want to share one tip you can try to avoid race conditions when saving a computed value.

Using the account balance as an example

To talk about something concrete I will use the “account balance” as an example, but you can use this approach for different types of calculations.

The account balance example

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

Tip: Pick a safe previous date

If your app is really concurrent and you can’t show posible off value, maybe you can pick a “safe previous date” like “yesterday” or “an hour ago” and present the value for that date.

You can run a daily rake task to save the balance with something like this:

class Account < ApplicationRecord
  has_many :entries

  def balance_at(datetime)
    entries.where(created_at: ..datetime).sum(:amount)
  end

  def update_safe_balance!
    self.safe_balance_time = 1.day.ago.end_of_day
    self.safe_balance = balance_at(safe_balance_time)
    save!
  end

  def update_safe_balance_later
    Account::UpdateSafeBalanceJob.perform_later(self)
  end
end

class Entry < ApplicationRecord
  belongs_to :account, touch: true
end

class UpdateSafeBalanceJob < ApplicationJob
  def perform(account)
    account.update_safe_balance!
  end
end

# on a daily task
Account.find_each(&:update_safe_balance_later)

And then you can use it in your views indicating the time of the calculation:

<p>
  Balance:
  <%= number_to_currency account.safe_balance %>
  <small>Updated <%= account.safe_balance_time %></small>
</p>

Are we really avoiding race conditions?

As far as I understand, yes! We are avoiding race conditions because we are calculating the balance for a “safe” point in time. What does “safe” means will depend on your use case.

Is this a perfect fix?

No, because you are not really showing the “current value”, and you will need to make sure that your users understand that this is a calculation for a previous point in time, and that can be confusing.

Reference

I learned this tip from recycledcoder on reddit.

Do you know other problems with this solution?

If you have experience with other problems with this solution, please leave a comment =)

Touch + fragment caching to avoid race conditions saving computed values

2023-01-31T23:16:00Z

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result due to race conditions.

I have already shared an exercise to help you get more sensitivity about when an implementation can save a wrong value due to race conditions.

Here I want to share one tip you can try to avoid race conditions when saving a computed value.

Using the account balance as an example

To talk about something concrete I will use the “account balance” as an example, but you can use this approach for different types of calculations.

The account balance example

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

Now imagine that each account will need to create many entries concurrently, maybe on different background jobs or different requests. So if you want to calculate the balance and save it just after an entry is created, you could have problems with race conditions.

Tip: Use touch + fragment caching

If you will use the calculated value mostly in your views, and you need a way of saving the value…

Instead of trying to save the value in the database when an entry is created, you can use touch on the association and try fragment caching.

class Account < ApplicationRecord
  has_many :entries

  def balance
    entries.sum(:amount)
  end
end

class Entry < ApplicationRecord
  belongs_to :account, touch: true
end

<%= cache(account) %>
  <p>Balance: <%= number_to_currency account.balance %></p>
<% end %>

If you need to use the value in a list you can use collection caching, and rails will read all cache templates at once instead of one by one.

<%= render partial: 'account', collection: @account, cached: true %>

Are we really avoiding race conditions?

With this “fix” you are still prune to race conditions, but you are delaying the problem until the value is used.

In my understanding even if you get into race conditions, it would not be that problematic, because even if an other entry is created just after the cache check, when the page is visited again the cache will be refreshed. I can’t see a moment where the cache can get stuck on a previous value.

Is this a perfect fix?

No, you can have other type of problems.

For example, now you have to deal with cache expiration, in the example the calculation just depends on the account.entries. We have an easy way to update the account via touch, but other calculation can have dependencies that are can make the cache expiration more complex.

Do you know other problems with this solution?

If you have experience with other problems with this solution, please leave a comment =)

Will it save the wrong value?

2023-01-03T23:18:00Z

I have already written about how you should be aware about race conditions when saving a computed value, because you could save a wrong value.

Here I want to share with you an exercise to help you (and me 😅) get more sensitivity about when an implementation can save a wrong value thanks to race conditions.

Your task

I will show you some excercises with different ways/methods to calculate the balance and for each method you will have to think if the implementation can save the wrong value due to race conditions or not.

I am presenting my answer for each problem as guide, but this kind of problems are hard for me also. So, if you see an error please tell me in the comments.

Problem description

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

class Account < ActiveRecord::Base
  has_many :entries

  def create_entry(amount:)
    # we are going to change what happen here
  end
end

class Entry < ActiveRecord::Base
  belongs_to :account
end

But the problem is that the account will need to create many entries concurrently, maybe on different background jobs or different requests.

To simulate concurrency you can use a code like this:

threads = 4.times.map do |i|
  Thread.new do
    Account.first.create_entry(amount: 100)
  end
end

threads.each(&:join)
Account.first.balance

Or you can use the tool described in the post: Examples to explore possible race conditions when caching custom computed values in Rails.

Exercise 00 - Add balance

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance_with(entry)
  end

  def update_balance_with(entry)
    balance = self.balance + entry.amount
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true
end

Answer

It can save the wrong value due to race conditions. You can see this example.

Exercise 01 - Sum balance in ruby

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(&:amount)
  end
end

Answer

It can save the wrong value due to race conditions. You can see this example.

Exercise 02 - Sum balance in database

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

It can save the wrong value due to race conditions. You can see this example.

Exercise 03 - Sum balance in database on after touch

class Account < ExampleRecord
  has_many :entries

  after_touch :update_balance

  def create_entry(amount:)
    entries.create(amount: amount)
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

As far as I understand, if you are using Postgres with the default isolation level Read Committed, you can compute and save the value on an after_touch.

Apparently after_touch works something like this:

There is a change in the association.
Within the same transaction, rails updates the associated record (account).
Then it performs the other operation that you have configured (update_balance).

PostgreSQL uses by default the Read Committed isolation level.

You can test more, with this example.

Exercise 04 - Sum balance in database using with_lock

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entries.create(amount: amount)
    update_balance
  end

  def update_balance
    with_lock do
      balance = entries.balance
      update!(balance: balance)
    end
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

In the example the calculation of the balance and the saving will be in a lock. So, at least this part will always be right.

But as we are dividing the process in two steps:

Creating the entry
Calculating and saving the balance

… it could happen that the first time that the balance is saved will be with a result different than 100, because more entries have been created before that moment.

You can check this run as an example of this problem:

Run with ThreadsTransaction (run 1)
[2] Record created: 77
[0] Record created: 76
[3] Record created: 75
[1] Record created: 78
[2] Balance calculated: 400
[2] Balance saved: 400
[1] Balance calculated: 400
[1] Balance saved: 400
[3] Balance calculated: 400
[3] Balance saved: 400
[0] Balance calculated: 400
[0] Balance saved: 400
Output: 400

Exercise 05 - Sum balance in database using with_lock on after touch

class Account < ExampleRecord
  has_many :entries
  after_touch :update_balance

  def create_entry(amount:)
    entries.create(amount: amount)
  end

  def update_balance
    with_lock do
      balance = entries.balance
      update!(balance: balance)
    end
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

Taking in account what I said in the exercises 03 and 04, the after touch wraps the call to “update_balance” in the same transaction of the entry creation. So, maybe the lock is not necessary.

You can test more, with this example.

Exercise 06 - Sum balance in database using with_lock on full operation

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    with_lock do
      entries.create(amount: amount)
      update_balance
    end
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

In this example we are wrapping each entry creation process on a lock. It, should always produce the right balance.

You can test more, with this example.

Exercise 07 - Updating counters

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance_with(entry)
  end

  def update_balance_with(entry)
    Account.update_counters(id, balance: entry.amount)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

Answer

This is very similar to the exercise 04, we are (kind of) dividing the process in two steps:

Creating the entry
Calculating and saving the balance

So, it could happen that the first time that the balance is saved will be with a result different than 100, because more entries have been created before that moment.

You can check this run as an example of this problem.

Saving incorrect computed values thanks to race conditions

2022-12-22T23:31:00Z

When saving computed values in the database in your rails app, you must be aware that is possible to find unexpected errors in the result thanks to race conditions.

Here I want to help you visualize how race conditions can make you save incorrect values even when your calculation is correct.

I will show you how it can happen with different ways of saving the same value and show you a tool that you can use to explore a little more.

Using the account balance as an example

Imagine that you have an Account record that has many entries, and you want to update the balance each time an Entry is created. The balance is the sum of the amount of each entry.

class Account < ActiveRecord::Base
  has_many :entries

  def create_entry(amount:)
    # we are going to change what happen here
  end
end

class Entry < ActiveRecord::Base
  belongs_to :account
end

Adding to the current state

One possible solution to calculate the balance, is start with zero and sum the amount of each new entry to the current balance.

Something like this:

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance_with(entry)
  end

  def update_balance_with(entry)
    balance = self.balance + entry.amount
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true
end

That looks ok, and if you write something like this…

account = Account.first
4.times { |i| account.create_entry(amount: 100) }
puts account.reload.balance

You can expect a printed balance of 400. If you run the code that will be correct.

But if instead of creating the entries serially, you create each entry in a different process or thread, concurrently, maybe on different background jobs, different requests or with code like this:


threads = 4.times.map do |i|
  Thread.new do
    Account.first.create_entry(amount: 100)
  end
end

threads.each(&:join)
Account.first.balance

Now the saved balance at the end of the process could have a wrong value.

… Can you see why?

If you don’t know why, don’t worry, it is not obvious at all!

Let’s try to understand why it can save a wrong value

Let’s start by breaking what is happening each time we create an entry.

Each time we call account.create_entry we:

Create the entry
Calculate the new balance
Save the balance

When we create all entries serially, each step happen like in the next code block:

# Run with SerialTransaction
# [0] Record created
# [0] Balance calculated: 100
# [0] Balance saved: 100
# [1] Record created
# [1] Balance calculated: 200
# [1] Balance saved: 200
# [2] Record created
# [2] Balance calculated: 300
# [2] Balance saved: 300
# [3] Record created
# [3] Balance calculated: 400
# [3] Balance saved: 400
# Output: 400

Each step is in the “expeted order”.

Each call to account.create_entry is represented by an index inside brackets, like [0], [1], etc.

But when we create the entries concurrently, we will see that the steps of each call to account.create_entry will be mixed.

For example the next code block is an example run of running each call to account.create_entry on a new thread:

# Run with ThreadsTransaction
# [3] Record created: 08
# [3] Balance calculated: 100
# [1] Record created: 09
# [1] Balance calculated: 100
# [0] Record created: 07
# [0] Balance calculated: 100
# [2] Record created: 10
# [2] Balance calculated: 100
# [0] Balance saved: 100
# [1] Balance saved: 100
# [3] Balance saved: 100
# [2] Balance saved: 100
# Output: 100

And as you can see the final output is 100 not 400!

If we see with care, we can see some funky things, like:

The records where not created in order
All balance calculations where 100
Every time we saved the balance we always saved 100

Why do you think that every balance calculation were 100 ?

The reason is that the four calls to account.create_entry, start at the same time and at that time the current balance is 0 for each call, then we “correctly” sum the entry.amount of 100 and the total was 100 every time.

Recalculating the value

But what if instead of using the currently saved balance, we recalculate the balance?

Maybe with something like this:

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(:amount)
  end
end

If we break what is happening each time we create an entry, we can see that we:

Create the entry
Calculate the new balance
Save the balance

Yes, just as in the previous example!

And when we run the example creating each entry on a different thread, we can see that it can also can have problems:

# Run with ThreadsTransaction
# [1] Record created: 23
# [1] Balance calculated: 100
# [2] Record created: 25
# [0] Record created: 26
# [2] Balance calculated: 300
# [3] Record created: 24
# [0] Balance calculated: 400
# [3] Balance calculated: 400
# [1] Balance saved: 100
# [0] Balance saved: 400
# [3] Balance saved: 400
# [2] Balance saved: 300
# Output: 300

Although sometimes we are going to get the right value… We can’t know when it is going to be wrong.

Be aware race conditions

Now (I hope) you understand that saving a computed value that can be updated by processes or threads running concurrently, maybe on different background jobs or different web requests, can cause race conditions and incorrect saved values.

How to solve these kind of problems?

I really don’t know 😅

There are different techniques that you can try but I don’t know a way that will always work and won’t give you troubles in other way.

Still, here are some options that you can try:

Don’t save the value
Use fragment caching
Pick a “safe” previous date
Use domain knowledge, to save the value when concurrency will not be present
Use database locks
Use after touch

A tool to explore more

I built a repo with the setup tor run the examples of this post (and some more), that can help you explore other solutions that could have errors and others that won’t.

You can get more details on the post: Examples to explore possible race conditions when caching custom computed values in Rails

Examples to explore possible race conditions when caching custom computed values in Rails

2022-12-07T23:42:00Z

In Rails, sometimes you will need to save counts or custom computed values, where the default counter cache will not be enough.

Maybe you want to…

Update a counter cache when a value change and not only when the association is created or deleted.
Have a counter cache for complex “has many through” associations.
Keep a count for a scope of the association, like just the “positive reactions”, or the “completed orders”.
Cache a sum or another calculation like the “account balance”.

There are different techniques for caching this kind of values, but sometimes this type of calculation are prone to race conditions.

Here I want share a tool to help you understand why caching this kind of values are prone to race conditions, analyzing different ways to solve the “account balance” problem. Hoping it could help you extrapolate to other situations.

A list of runnable examples

The tool that I am talking about is a list of runnable examples that you can find in the repo: github.com/bhserna/custom_computed_values_and_race_conditions

How can this examples help you?

You will be able to compare the threads of execution for each example with an output like this:

# Run with ThreadsTransaction
# [1] Record created: 15
# [1] Balance calculated: 100
# [2] Record created: 16
# [2] Balance calculated: 200
# [3] Record created: 17
# [3] Balance calculated: 300
# [0] Record created: 18
# [0] Balance calculated: 400
# [2] Balance saved: 200
# [0] Balance saved: 400
# [1] Balance saved: 100
# [3] Balance saved: 300
# Output: 300

Each example will run a “transaction” (not db transaction) where it will create four entries in an account with an amount of 100 and calculate the balance of the account after the creation of each entry. Expecting a final balance of 400.

Each example calculates the balance in a slightly different way. For example one adds the amount of each entry to the current account.balance, like this:

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance_with(entry)
  end

  def update_balance_with(entry)
    balance = self.balance + entry.amount
    update!(balance: balance)
  end
end

And other sums the amount of all account.entries with ruby, each time that an entry is created, like this:

class Account < ExampleRecord
  has_many :entries

  def create_entry(amount:)
    entry = entries.create(amount: amount)
    update_balance
  end

  def update_balance
    balance = entries.balance
    update!(balance: balance)
  end
end

class Entry < ExampleRecord
  belongs_to :account, touch: true

  def self.balance
    sum(&:amount)
  end
end

In other examples the sum is done in the database, other examples use with_lock in different places, and other examples use the class method ExampleRecord.update_counters from rails.

Each example will run the “transaction” using three different strategies:

SerialTransaction - That will create the entries serially in the same thread.
ThreadsTransaction - That will create each entry and update the balance in a different thread.
ForksTransaction - That will create each entry and update the balance in a different process, using fork.

For the ThreadsTransaction and ForksTransactions it will run the transaction until a run returns a different balance than the expected of 400, or if the number of runs equals the parameter max_runs_per_transaction_type that has 5 as default value.

When you run an example you will find an output like:

# Run with SerialTransaction
# [0] Record created: 03
# [0] Balance calculated: 100
# [0] Balance saved: 100
# [1] Record created: 04
# [1] Balance calculated: 200
# [1] Balance saved: 200
# [2] Record created: 05
# [2] Balance calculated: 300
# [2] Balance saved: 300
# [3] Record created: 06
# [3] Balance calculated: 400
# [3] Balance saved: 400
# Output: 400
#
# Run with ThreadsTransaction
# [1] Record created: 07
# [2] Record created: 08
# [1] Balance calculated: 100
# [0] Record created: 09
# [2] Balance calculated: 300
# [3] Record created: 10
# [0] Balance calculated: 300
# [3] Balance calculated: 400
# [1] Balance saved: 100
# [2] Balance saved: 300
# [0] Balance saved: 300
# [3] Balance saved: 400
# Output: 400
#
# Run with ThreadsTransaction
# [1] Record created: 11
# [1] Balance calculated: 100
# [2] Record created: 12
# [2] Balance calculated: 200
# [3] Record created: 13
# [3] Balance calculated: 300
# [0] Record created: 14
# [0] Balance calculated: 400
# [2] Balance saved: 200
# [3] Balance saved: 300
# [1] Balance saved: 100
# [0] Balance saved: 400
# Output: 400
#
# Run with ThreadsTransaction
# [1] Record created: 15
# [1] Balance calculated: 100
# [2] Record created: 16
# [2] Balance calculated: 200
# [3] Record created: 17
# [3] Balance calculated: 300
# [0] Record created: 18
# [0] Balance calculated: 400
# [2] Balance saved: 200
# [0] Balance saved: 400
# [1] Balance saved: 100
# [3] Balance saved: 300
# Output: 300
#
# Run with ForksTransaction
# [2] Record created: 20
# [1] Record created: 19
# [2] Balance calculated: 200
# [0] Record created: 21
# [1] Balance calculated: 300
# [0] Balance calculated: 300
# [3] Record created: 22
# [3] Balance calculated: 400
# [3] Balance saved: 400
# [2] Balance saved: 200
# [0] Balance saved: 300
# [1] Balance saved: 300
# Output: 300

To explain what that means, we can zoom in to the output of the last run in the ThreadsTransaction…

# Run with ThreadsTransaction
# [1] Record created: 15
# [1] Balance calculated: 100
# [2] Record created: 16
# [2] Balance calculated: 200
# [3] Record created: 17
# [3] Balance calculated: 300
# [0] Record created: 18
# [0] Balance calculated: 400
# [2] Balance saved: 200
# [0] Balance saved: 400
# [1] Balance saved: 100
# [3] Balance saved: 300
# Output: 300

The number in brackets ([]) is an index of the current thread of execution of each created entry. So you will be able to compare how each part of the process is executed.

For example in the previous log we can see some, maybe unexpected, things like:

The thread [0] created the record after the other three threads.
The thread [3] calculates the balance before the last record is created, but is the last that saves the balance, giving us a bad result at the end.

How to run the examples

Install the dependencies with bundle install.
Database setup - run the command:

ruby db/setup.rb

Run the examples with ruby examples/<file name>. For example:

ruby example/00_example.rb

Change the seeds on db/seeds.rb and re-run ruby db/setup.rb to test different scenarios.

Broadcast multiple actions with broadcast_render

2022-09-19T22:28:00Z

Imagine that you are creating a record (let’s say a tweet), and on creation, you want to add the tweet to the page and also update the tweets count in the page…

But you want to update the content, not just for the current user, but for all the users in the page, you want to broadcast the changes.

How would you do it?

Well, here I want to show you how you can use broadcast_render or its _later variations, to do it.

Example video

You will be able to do something like this:

Use broadcast_render_later_to

This method will help you render a turbo stream template with the current model as the local variable, so you can send something like this:

<%= turbo_stream.replace "new_tweet" do %>
  <%= render "tweets/form", tweet: Tweet.new %>
<% end %>

<%= turbo_stream.prepend "tweets", tweet %>

<%= turbo_stream.replace "tweets_count" do %>
  <%= render "tweets/count", tweets: Tweet.all %>
<% end %>

If you do:

tweet.broadcast_render_later_to :tweets, partial: "tweets/on_create"

Why you should prefer broadcast_render_later_to over broadcast_render_to?

As normally these methods will render some templates and the send them, if you use broadcast_render_to as a reaction to a controller action, you will block the response of the controller action until the rendering is done.

Instead if you use broadcast_render_later_to the rendering and broadcast would be executed on a background job, and won’t block the controller’s response.

When you are destroying a record (as far as I understand) you won’t be able to use broadcast_render_later_to because as the record is already destroyed, it won’t be able to fetch the record on the job. So, after destroy, you could use broadcast_render_to.

Let’s see the full example

For the example of the video…

You can have a tweets/index like this:

<%= turbo_stream_from :tweets %>

<header>
  <%= render "count", tweets: @tweets %>
  <h1>Tweets</h1>
  <%= render "form", tweet: Tweet.new %>
</header>

<section id="tweets">
  <%= render @tweets %>
</section>

It needs the turbo_stream_from :tweets to listen to the broadcasts that you will produce.

The form could be a “standard” form like:

<%= form_with(model: tweet, id: dom_id(tweet)) do |form| %>
  <div>
    <%= form.text_area :body %>
  </div>

  <div>
    <%= form.submit %>
  </div>
<% end %>

Just look that in this case I am using dom_id(tweet) as the form id. In that way, for non persisted records, the id will be new_tweet and it could be replaced with turbo_stream.replace "new_tweet".

Then in the controller you can create the tweet and do the broadcast there, without any other response.

def create
  @tweet = Tweet.new(tweet_params)
  @tweet.save
  @tweet.broadcast_render_later_to :tweets, partial: "tweets/on_create"
end

The partial tweets/on_create.turbo_stream.erb could look something like this:

<%= turbo_stream.replace "new_tweet" do %>
  <%= render "tweets/form", tweet: Tweet.new %>
<% end %>

<%= turbo_stream.prepend "tweets", tweet %>

<%= turbo_stream.replace "tweets_count" do %>
  <%= render "tweets/count", tweets: Tweet.all %>
<% end %>

It will:

Replace the form with a new form to clear the textarea
Preprend the new tweet
Update the “tweets count”

If you think that is better for your use case to render from the model, you can do it on an after_create_commit:

class Tweet < ApplicationRecord
  after_create_commit do
    broadcast_render_later_to :tweets, partial: "tweets/on_create"
  end
end

Play with the code

Here is code of the example app that I used for the video and the examples, go and play with it.

github.com/bhserna/broadcast_render_example

Building a dynamic data grid with search and filters using rails, hotwire and ransack

2022-09-03T12:24:00Z

Do you want to build powerful admin interfaces with little code, but you are not sure if you want to jump into a full admin solution like Active Admin, Administrate or Avo?

Here I want to show you an alternative!

A step by step guide to build a dynamic data grid with search and filters for your admin interfaces, using rails, the ransack gem and hotwire.

What are you going to build?

You are goint to build a paginated table with four filters and sortable columns.

Something like this:

Step 1: A non-dynamic paginated table

👉 You can find the full commit here.

The first step is a common rails index with the pagy gem to build the pagination.

Create the record

For the example we are going to use three columns name, category and price.

class CreateProducts < ActiveRecord::Migration[7.0]
  def change
    create_table :products do |t|
      t.string :name
      t.integer :category
      t.decimal :price

      t.timestamps
    end
  end
end

And category is going to be an enum:

class Product < ApplicationRecord
  enum :category, [:toys, :electronics, :food]
end

Declare the routes

We are going to create a products resource with only the index route.

And to make it easy to test we can define the root as a redirect to /products.

Rails.application.routes.draw do
  root to: redirect("/products")
  resources :products, only: [:index]
end

The controller

For this phase of the project we are going to paginate all products, with 10 items per page.

Once you have the setup for the pagy gem, the controller code will look like this:

class ProductsController < ApplicationController
  def index
    @pagy, @products = pagy(Product.all, items: 10)
  end
end

And then add the views…

The index.html.erb:

<h1>Products</h1>

<div class="table-container">
  <table>
    <thead>
      <tr>
        <th>Name</th>
        <th>Category</th>
        <th>Price</th>
      </tr>
    </thead>
    <tbody>
      <%= render @products %>
    </tbody>
  </table>
</div>

<%= raw pagy_nav(@pagy) %>

And the _product.html.erb partial

<%= tag.tr id: dom_id(product) do %>
  <td><%= product.name %></td>
  <td><%= product.category %></td>
  <td><%= number_to_currency product.price %></td>
<% end %>

Enough CSS

To make it look descent you can put this css on your application.css file:

@import "https://unpkg.com/open-props";

* { box-sizing: border-box; }

body {
  font-family: var(--font-sans);
  font-size: var(--font-size-1);
  margin-right: auto;
  margin-left: auto;
  padding: var(--size-5);
}

.table-container { overflow: auto; }
table { width: 100%; }
tbody { line-height: var(--font-lineheight-3); }
td, th {
  text-align: left;
  padding-bottom: var(--size-2);
  padding-right: var(--size-2);
  background-color: white;
  border-color: var(--gray-3);
  border-bottom-style: solid;
  border-bottom-width: var(--border-size-1);
}
th { font-weight: 600; }
td { padding-top: var(--size-2); }

.pagy-nav { padding: var(--size-7) 0; }
.pagy-nav .page { padding: 0 var(--size-1); }

Step 2: Order columns with sort_link

👉 You can find the full commit here.

To make the columns sortable with ransack you need to use the helper sort_link on each th from the table:

<thead>
  <tr>
    <th><%= sort_link(@q, :name) %></th>
    <th><%= sort_link(@q, :category) %></th>
    <th><%= sort_link(@q, :price) %></th>
  </tr>
</thead>

And then use the ransack method to create a Ransack::Search that you should assign to the instance variable @q.

def index
  @q = Product.ransack(params[:q])
  scope = @q.result(distinct: true)
  @pagy, @products = pagy(scope, items: 10)
end

If you want to set a default sort you can do something like this:

@q.sorts = "name asc" if @q.sorts.empty?

In the example app I put the code in a private method like this:

def index
  @q = Product.ransack(params[:q])
  scope = @q.result(distinct: true)
  set_default_sort
  @pagy, @products = pagy(scope, items: 10)
end

private

def set_default_sort
  @q.sorts = "name asc" if @q.sorts.empty?
end

Step 3: Search with ransack

👉 You can find the full commit here.

To search with ransack you need to follow the convention defined in the documentation.

In the documentation you can also find a list with all possible matchers/predicates.

In the example we have four filters:

A search_field for name_cont, that match if the name contains the given value.
A select for category_eq with the list of possible categories, that match if the category equals the selected value.
A search_field for price_gt, that match if the price is greater than the given value.
A search_field for price_lt, that match if the price is less than the given value.

To implement it you need to write this erb.

<%= search_form_for @q do |f| %>
  <div>
    <%= f.label :name_cont %>
    <%= f.search_field :name_cont %>
  </div>

  <div>
    <%= f.label :category_eq %>
    <%= f.select :category_eq, Product.categories, include_blank: true %>
  </div>

  <div>
    <%= f.label :price_gt %>
    <%= f.search_field :price_gt %>
  </div>

  <div>
    <%= f.label :price_lt %>
    <%= f.search_field :price_lt %>
  </div>

  <div>
    <%= f.submit %>
  </div>
<% end %>

To style the form you can use this css:

 label {
   display: block;
 }

 input[type="search"],
 select {
   -webkit-appearance: none;
   -moz-appearance: none;
   background-color: white;
   border: var(--gray-3) solid var(--border-size-1);
   border-radius: var(--radius-2);
   margin-top: var(--size-2);
   padding: var(--size-2);
   height: var(--size-8);
   display: block;
   font-size: var(--font-size-1);
   width: 100%;
 }

 input[type="submit"] {
   padding: var(--size-2) var(--size-4);
   border: var(--blue-6) solid var(--border-size-1);
   background-color: var(--blue-6);
   color: white;
   border-radius: var(--radius-2);
   font-size: var(--font-size-1);
 }

 input[type="submit"]:hover {
   background-color: var(--blue-8);
 }

 form {
   display: grid;
   grid-template-columns: repeat(5, 1fr);
   gap: var(--size-2);
   align-items: flex-end;
 }

 .table-container {
   margin-top: var(--size-7);
   overflow: auto; 
 }

Step 4: Using turbo to send the form when somthing changes

👉 You can find the full commit here.

Now you can remove the button and instead send the form when something in it changes.

To do it you can create an stimulus controller that calls “requestSubmit()” when something changes.

It could be something like this:

// filters_controller.js
import { Controller } from "@hotwired/stimulus"

 export default class extends Controller {
   submit(event) {
     this.element.requestSubmit()
   }
 }

Now you can update the form to define the controller and to reference a turbo frame to update. Like this:

<%= search_form_for @q, html: {"data-controller": "filters", "data-turbo-frame": "table"} do |f| %>

Then on each control you should define an action that will call “filters#submit” and remove the submit button.

<%= search_form_for @q, html: {"data-controller": "filters", "data-turbo-frame": "table"} do |f| %>
  <div>
    <%= f.label :name_cont %>
    <%= f.search_field :name_cont, "data-action": "filters#submit" %>
  </div>

  <div>
    <%= f.label :category_eq %>
    <%= f.select :category_eq, Product.categories, {include_blank: true}, {"data-action": "filters#submit"} %>
  </div>

  <div>
    <%= f.label :price_gt %>
    <%= f.search_field :price_gt, "data-action": "filters#submit" %>
  </div>

  <div>
    <%= f.label :price_lt %>
    <%= f.search_field :price_lt, "data-action": "filters#submit" %>
  </div>
<% end %>

You will need to wrap the table and pagination in a turbo_frame_tag that match what you defined in the form, in this case you need to call it "table":

<%= turbo_frame_tag "table" do %>
  <table>
    <thead>
      <tr>
        <th><%= sort_link(@q, :name) %></th>
        <th><%= sort_link(@q, :category) %></th>
        <th><%= sort_link(@q, :price) %></th>
      </tr>
    </thead>
    <tbody>
      <%= render @products %>
    </tbody>
  </table>

  <%= raw pagy_nav(@pagy) %>
<% end %>

You can update the css of your form to take the full width and give the same space to each control:

form {
  display: grid;
  grid-auto-flow: column;
  grid-auto-columns: 1fr;
  gap: var(--size-2);
  align-items: flex-end;
}

Step 6: Adding a “loading” indicator

👉 You can find the full commit here.

To add a loading indicator you can use the attribute busy on the turbo-frame.

It is a boolean attribute managed by turbo. It is toggled to be present when a turbo-frame-initiated request starts, and toggled false when the request ends.

Then you can add an tag with the loading class on the turbo frame:

<%= turbo_frame_tag "table" do %>
  <p class="loading">Loading...</p>
  ...
<% end %>

And add some css to toggle the visibility based on the busy attribute:

.loading { visibility: hidden; }
turbo-frame[busy] .loading { visibility: visible; }

Summary

This is the code that you need to build a dynamic paginated data-grid with ransack and hotwire.

On the controller you need to:

Call ransack with params[:q]
Pass the result to pagy
And if you want, you can set a default order

class ProductsController < ApplicationController
  def index
    @q = Product.ransack(params[:q])
    scope = @q.result(distinct: true)
    set_default_sort
    @pagy, @products = pagy(scope, items: 10)
  end

  private

  def set_default_sort
    @q.sorts = "name asc" if @q.sorts.empty?
  end 
end

Then on the view you:

Pass @q to the serch_form_for helper.
Define the filters controller on the form.
Tell the form to act on the table turbo frame.
Define an action to filters#submit on each input.
Add the sort_link on each th that you want.

<h1>Products</h1>

<%= search_form_for @q, html: {"data-controller": "filters", "data-turbo-frame": "table"} do |f| %>
  <div>
    <%= f.label :name_cont %>
    <%= f.search_field :name_cont, "data-action": "filters#submit" %>
  </div>

  <div>
    <%= f.label :category_eq %>
    <%= f.select :category_eq, Product.categories, {include_blank: true}, {"data-action": "filters#submit"} %>
  </div>

  <div>
    <%= f.label :price_gt %>
    <%= f.search_field :price_gt, "data-action": "filters#submit" %>
  </div>

  <div>
    <%= f.label :price_lt %>
    <%= f.search_field :price_lt, "data-action": "filters#submit" %>
  </div>
<% end %>

<%= turbo_frame_tag "table" do %>
  <p class="loading">Loading...</p>

  <table>
    <thead>
      <tr>
        <th><%= sort_link(@q, :name) %></th>
        <th><%= sort_link(@q, :category) %></th>
        <th><%= sort_link(@q, :price) %></th>
      </tr>
    </thead>
    <tbody>
      <%= render @products %>
    </tbody>
  </table>

  <%= raw pagy_nav(@pagy) %>
<% end %>

To submit the form on filters#submit you need a controller like this:

import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  submit(event) {
    this.element.requestSubmit()
  }
}

Fetching the top n per group with window functions

2022-08-22T22:34:00Z

Have you ever needed to get the most recent N posts for each user in rails, but didn’t know how to do it without using map?

Or maybe something similar like:

The first or last X comments for each post
The first or last Y payments for each customer
The first or last Z reviews for each customer

Sometimes could be ok to just fetch all elements and filter with ruby, but sometimes it is not possible. Also it can cause n+1 queries if your are not careful.

Here I want to show you how you can solve this problem using window functions.

An example problem

Imagine you have a User and a Post record and a User has many Posts, like this:

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

And you want to fetch a list of users with their last 3 posts, but for performance reasons you don’t want to fetch all posts for all users in the list, you just want to ask the database for the 3 posts per user that you want to use.

Building the SQL query

First we need to order the posts and produce a rank for each user.

SELECT posts.*, dense_rank() OVER (
  PARTITION BY posts.user_id
  ORDER BY posts.id DESC
) AS posts_rank
FROM posts

And then use that query as a subquery to now select just the posts with rank less than or equals to 3.

SELECT * FROM (
  SELECT posts.*, dense_rank() OVER (
    PARTITION BY posts.user_id
    ORDER BY posts.id DESC
  ) AS posts_rank
  FROM posts
) AS ranked_posts
WHERE posts_rank <= 3

This will return a list with just the last three posts for each user.

Scope with a fixed n

Now we need to pass this query to rails in a way that it could build Post objects from this information.

This is how I made it work.

class Post < ActiveRecord::Base
  scope :last_n_per_user, -> {
    select_sql = <<-SQL
      posts.*, dense_rank() OVER (
        PARTITION BY posts.user_id
        ORDER BY posts.id DESC
      ) AS posts_rank
    SQL

    ranked_posts = select(select_sql)
    from(ranked_posts, "posts").where("posts_rank <= 3")
  }
end

From the sql we need the part that selects the posts with the rank, to then use it in the rails select method.

Then we use from to use the ranked_posts relation as subquery and then use where to define the post_rank that we want to filter.

The post_rank is what defines the “n”. The number of posts per user that we want to fetch.

At the end the sql it builds is very similar to our original sql:

SELECT posts.*
FROM (
  SELECT posts.*, dense_rank() OVER (
    PARTITION BY posts.user_id
    ORDER BY posts.id DESC
  ) AS posts_rank
  FROM posts
) posts
WHERE posts_rank <= 3

Scope with a variable n

If you want to make the “n” variable, you can pass it as a parameter to the where method:

class Post < ActiveRecord::Base
  scope :last_n_per_user, ->(n) {
    select_sql = <<-SQL
      posts.*, dense_rank() OVER (
        PARTITION BY posts.user_id
        ORDER BY posts.id DESC
      ) AS posts_rank
    SQL

    ranked_posts = select(select_sql)
    from(ranked_posts, "posts").where("posts_rank <= ?", n)
  }
end

Put it in a has_many association

Now that you have build the scope, you can build a has_many association passing that scope, like this:

class User < ActiveRecord::Base
  has_many :posts
  has_many :last_posts, -> { last_n_per_user(3) },
    class_name: "Post"
end

And now you will be able to preload just the last_posts like this:

users = User.preload(:last_posts).limit(5)

users.each do |user|
  puts user.last_posts.map(&:id).inspect
end

Code to run the example

Here is the full code that you can use to run with the Active Record Playground Runner.

schema do
  create_table :users do |t|
    t.string :name
  end

  create_table :posts do |t|
    t.integer :user_id
    t.string :title
  end

  add_index :posts, :user_id
end

seeds do
  users = create_list(User, count: 10) do
    { name: FFaker::Name.name }
  end

  create_list_for_each_record(Post, records: users, count: 100) do |user|
    { user_id: user.id, title: FFaker::CheesyLingo.title }
  end
end

models do
  class User < ActiveRecord::Base
    has_many :posts
  end

  class Post < ActiveRecord::Base
    belongs_to :user
  end
end

example "SQL query" do
  sql = <<-SQL
    SELECT * FROM (
      SELECT posts.*, dense_rank() OVER (
        PARTITION BY posts.user_id
        ORDER BY posts.id DESC
      ) AS posts_rank
      FROM posts
    ) AS ranked_posts
    WHERE posts_rank <= 3
  SQL

  result = ActiveRecord::Base.connection.execute(sql)
  puts result.to_a.inspect
end

example "With a fixed n" do
  class Post < ActiveRecord::Base
    scope :last_n_per_user, -> {
      select_sql = <<-SQL
        posts.*, dense_rank() OVER (
          PARTITION BY posts.user_id
          ORDER BY posts.id DESC
        ) AS posts_rank
      SQL

      ranked_posts = select(select_sql)
      from(ranked_posts, "posts").where("posts_rank <= 3")
    }
  end

  posts = Post.last_n_per_user.preload(:user)
  pp posts.group_by(&:user).map { |user, posts| [user.name, posts.map(&:id)] }
end

example "With a variable n" do
  class Post < ActiveRecord::Base
    scope :last_n_per_user, ->(n) {
      select_sql = <<-SQL
        posts.*, dense_rank() OVER (
          PARTITION BY posts.user_id
          ORDER BY posts.id DESC
        ) AS posts_rank
      SQL

      ranked_posts = select(select_sql)
      from(ranked_posts, "posts").where("posts_rank <= ?", n)
    }
  end

  posts = Post.last_n_per_user(3).preload(:user)
  pp posts.group_by(&:user).map { |user, posts| [user.name, posts.map(&:id)] }
end

example "In a has many association" do
  class User < ActiveRecord::Base
    has_many :posts
    has_many :last_posts, -> { last_n_per_user(3) }, class_name: "Post"
  end

  class Post < ActiveRecord::Base
    scope :last_n_per_user, ->(n) {
      select_sql = <<-SQL
        posts.*, dense_rank() OVER (
          PARTITION BY posts.user_id
          ORDER BY posts.id DESC
        ) AS posts_rank
      SQL

      ranked_posts = select(select_sql)
      from(ranked_posts, "posts").where("posts_rank <= ?", n)
    }
  end

  users = User.preload(:last_posts).limit(5)
  pp users.map { |user| [user.name, user.last_posts.map(&:id)] }
end

You can find the code also on: github.com/bhserna/last_n_per_user_window_functions

Fetching the top n per group with a lateral join with rails

2022-08-16T22:29:00Z

Have you ever needed to get the most recent N posts for each user in rails, but didn’t know how to do it?

Or maybe something similar like:

The first or last X comments for each post
The first or last Y payments for each customer
The first or last Z reviews for each customer

Sometimes it could be ok to just fetch all elements and filter with ruby, but sometimes it is not possible. Also it can cause n+1 queries if your are not careful.

Here I want to show you how you can solve this problem using a lateral join.

An example problem

Imagine you have a User and a Post record and a User has many Posts, like this:

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

Building the SQL query

We need to select from users and build a lateral join to a subquery that will select from posts but filtering the posts where the posts.user_id is the users.id and then order those posts to be able to limit the selection to just the 3 posts that we want to fetch.

SELECT selected_posts.* FROM users JOIN LATERAL (
  SELECT * FROM posts
  WHERE user_id = users.id
  ORDER BY id DESC LIMIT 3
) AS selected_posts ON TRUE

This will return as selected_posts a list with just the last three posts for each user.

Lateral join with rails with a fixed n

Now we need to pass this query to rails in a way that it could build Post objects from this information.

This is how I made it work.

class Post < ActiveRecord::Base
  belongs_to :user

  scope :last_n_per_user, -> {
    sql = <<-SQL
      JOIN LATERAL (
        SELECT * FROM posts
        WHERE user_id = users.id
        ORDER BY id DESC LIMIT 3
      ) AS selected_posts ON TRUE
    SQL

    selected_posts = User
      .select("selected_posts.*")
      .joins(sql)

    from(selected_posts, "posts")
  }
end

We start the query from the User model and then select the selected_posts. Then comes the lateral join.

Then we use Post.from to tell rails to evaluate the sql as a subquery and with it build the Post objects.

At the end it builds this sql:

SELECT "posts".* FROM (SELECT selected_posts.* FROM "users"
JOIN LATERAL (
  SELECT * FROM posts
  WHERE user_id = users.id
  ORDER BY id DESC LIMIT 3
) AS selected_posts ON TRUE) posts

Lateral join with rails with a variable n

If you want to make the n variable, you can use the sanitize the sql like this:

class Post < ActiveRecord::Base
  belongs_to :user

  scope :last_n_per_user, ->(n) {
    sql = <<-SQL
      JOIN LATERAL (
        SELECT * FROM posts
        WHERE user_id = users.id
        ORDER BY id DESC LIMIT :limit
      ) AS selected_posts ON TRUE
    SQL

    selected_posts = User
      .select("selected_posts.*")
      .joins(User.sanitize_sql([sql, limit: n]))

    from(selected_posts, "posts")
  }
end

Put it in a has_many association

Now that you have build the scope, you can build a has many association passing that scope, like this:

class User < ActiveRecord::Base
  has_many :posts
  has_many :last_posts, -> { last_n_per_user(3) },
    class_name: "Post"
end

And now you will be able to preload just the last_posts like this:

users = User.preload(:last_posts).limit(5)

users.each do |user|
  puts user.last_posts.map(&:id).inspect
end

Code to run the example

Here is the full code that you can use to run with the Active Record Playground Runner.

schema do
  create_table :users do |t|
    t.string :name
  end

  create_table :posts do |t|
    t.integer :user_id
    t.string :title
  end

  add_index :posts, :user_id
end

seeds do
  users = create_list(User, count: 10) do
    { name: FFaker::Name.name }
  end

  create_list_for_each_record(Post, records: users, count: 100) do |user|
    { user_id: user.id, title: FFaker::CheesyLingo.title }
  end
end

models do
  class User < ActiveRecord::Base
    has_many :posts
  end

  class Post < ActiveRecord::Base
    belongs_to :user
  end
end

example "SQL query" do
  sql = <<-SQL
    SELECT selected_posts.* FROM users JOIN LATERAL (
      SELECT * FROM posts
      WHERE user_id = users.id
      ORDER BY id DESC LIMIT 3
    ) AS selected_posts ON TRUE
  SQL

  result = ActiveRecord::Base.connection.execute(sql)
  puts result.to_a.inspect
end

example "Lateral join with rails with a fixed n" do
  class Post < ActiveRecord::Base
    scope :last_n_per_user, -> {
      sql = <<-SQL
        JOIN LATERAL (
          SELECT * FROM posts
          WHERE user_id = users.id
          ORDER BY id DESC LIMIT 3
        ) AS selected_posts ON TRUE
      SQL

      selected_posts = User
        .select("selected_posts.*")
        .joins(sql)

      from(selected_posts, "posts")
    }
  end

  posts = Post.last_n_per_user.preload(:user)
  pp posts.group_by(&:user).map { |user, posts| [user.name, posts.map(&:id)] }
end

example "Lateral join with rails with a variable n" do
  class Post < ActiveRecord::Base
    scope :last_n_per_user, ->(n) {
      sql = <<-SQL
        JOIN LATERAL (
          SELECT * FROM posts
          WHERE user_id = users.id
          ORDER BY id DESC LIMIT :limit
        ) AS selected_posts ON TRUE
      SQL

      selected_posts = User
        .select("selected_posts.*")
        .joins(User.sanitize_sql([sql, limit: n]))

      from(selected_posts, "posts")
    }
  end

  posts = Post.last_n_per_user(3).preload(:user)
  pp posts.group_by(&:user).map { |user, posts| [user.name, posts.map(&:id)] }
end

example "In a has many association" do
  class User < ActiveRecord::Base
    has_many :posts
    has_many :last_posts, -> { last_n_per_user(3) }, class_name: "Post"
  end

  class Post < ActiveRecord::Base
    scope :last_n_per_user, ->(n) {
      sql = <<-SQL
        JOIN LATERAL (
          SELECT * FROM posts
          WHERE user_id = users.id
          ORDER BY id DESC LIMIT :limit
        ) AS selected_posts ON TRUE
      SQL

      selected_posts = User
        .select("selected_posts.*")
        .joins(User.sanitize_sql([sql, limit: n]))

      from(selected_posts, "posts")
    }
  end

  users = User.preload(:last_posts).limit(5)
  pp users.map { |user| [user.name, user.last_posts.map(&:id)] }
end

You can find the code also on: github.com/bhserna/last_n_per_user_lateral_join

Active Record Playground Runner Introduction

2022-08-08T22:53:00Z

Have you ever wanted to just create an active record example to someone in your team without thinking in the database setup?

Or maybe send two different models designs with some examples to see the difference, but the setup what just to difficult?

Or like me, create a bunch of examples to teach an Active Record concept?

If you have had one of this problems, maybe this tool can help you.

A tool to run Active Record examples

I want to introduce a new tool that I am working on.

It will help you play with Active Record and Postgres, without the thinking in the database setup.

You will be able to declare the schema, models, seeds and examples in just one file, like this:

schema do
  create_table :posts do |t|
    t.string :title
    t.text :body
  end

  create_table :comments do |t|
    t.text :body
    t.integer :post_id
  end

  add_index :comments, :post_id
end

models do
  class Post < ActiveRecord::Base
    has_many :comments
  end

  class Comment < ActiveRecord::Base
    belongs_to :post
  end
end

seeds do
  posts = create_list(Post, count: 10) do
    {
      title: FFaker::CheesyLingo.title,
      body: FFaker::CheesyLingo.paragraph
    }
  end

  create_list_for_each_record(Comment, records: posts, count: 20) do |post|
    {
      post_id: post.id,
      body: FFaker::CheesyLingo.sentence
    }
  end
end

example "first" do
  Post.includes(:comments).limit(5).map do |post|
    puts post.comments.count
  end
end

example "second" do
  Post.includes(:comments).limit(5).map do |post|
    puts post.comments.size
  end
end

And the run it with:

bundle exec run_playground file_name.rb

And it will:

Create the database with a random name
Load the schema
Run the seeds
Run the examples with a nice format and with the logger turned on.
Drop the database

And print an output like:

Setup
-----
-- create_table(:posts)
   -> 0.3001s
-- create_table(:comments)
   -> 0.0092s
-- add_index(:comments, :post_id)
   -> 0.0041s


Example: first
--------------
D, [2022-08-08T18:13:46.118881 #89680] DEBUG -- :   Post Load (4.5ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
D, [2022-08-08T18:13:46.143351 #89680] DEBUG -- :   Comment Load (1.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5)  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4], ["post_id", 5]]
D, [2022-08-08T18:13:46.156379 #89680] DEBUG -- :   Comment Count (3.8ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
20
D, [2022-08-08T18:13:46.158504 #89680] DEBUG -- :   Comment Count (1.0ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 2]]
20
D, [2022-08-08T18:13:46.161012 #89680] DEBUG -- :   Comment Count (1.3ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 3]]
20
D, [2022-08-08T18:13:46.163467 #89680] DEBUG -- :   Comment Count (0.5ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 4]]
20
D, [2022-08-08T18:13:46.165299 #89680] DEBUG -- :   Comment Count (1.0ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 5]]
20


Example: second
---------------
D, [2022-08-08T18:13:46.166206 #89680] DEBUG -- :   Post Load (0.3ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
D, [2022-08-08T18:13:46.167795 #89680] DEBUG -- :   Comment Load (0.7ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5)  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4], ["post_id", 5]]
20
20
20
20
20

Why this tool?

I prepared a lot of examples for the ebook Avoid n+1 queries on rails using the first version of my Active Record Playground. A template to help in the setup of an environment with Active Record.

With the old template it was is hard to write a lot of examples with the same database schema, and it was hard to setup a project for each schema.

With this tool is much more easier to create examples with different schemas because you don’t need to setup a whole project directory for each schema.

It makes it easier to create more and more diverse examples. And also helping the people that see the examples to focus just on the example code instead of the project setup.

The code repository

You can find the code on: https://github.com/bhserna/active_record_playground_runner

Installation

For now, the gem is only on github, and if you want to install it you will need to add this to your Gemfile:

gem 'active_record_playground_runner', "~> 0.1.0", github: "bhserna/active_record_playground_runner", branch: "main"

How to use it

Schema

In the schema you will be able to define a schema using the methods that rails uses to define the schema in a regular rails application.

Models

Here you can define the records or plain old ruby objects that you will need on the seeds.

Seeds

You can use your models to create the state you need for your examples.

You will be able to use the FFaker gem. It will be already required.

Seeds Helpers

In the seeds from the example, there are two methods that are part of the gem, create_list and create_list_for_each_record.

seeds do
  posts = create_list(Post, count: 10) do
    {
      title: FFaker::CheesyLingo.title,
      body: FFaker::CheesyLingo.paragraph
    }
  end

  create_list_for_each_record(Comment, records: posts, count: 20) do |post|
    {
      post_id: post.id,
      body: FFaker::CheesyLingo.sentence
    }
  end
end

create_list receives a record class, a count keyword and a block. The block will be executed to create each record the number of times specified by count.

create_list_for_each_record also expects a list of records and for each record it will run the block count times.

Both methods will first build data as an array of hashes, and then use insert_all to create the records. Be aware that if you use this methods the callbacks will not be fired.

Examples

You can define as many examples as you want, you can pass a name if you need it:

example do
  Post.includes(:comments).limit(5).map do |post|
    puts post.comments.count
  end
end

example "My example" do
  Post.includes(:comments).limit(5).map do |post|
    puts post.comments.size
  end
end

If you need to define changes to the model classes, you will be able to do it inside the example:

example "third" do
  class Post < ActiveRecord::Base
    has_one :latest_comment, -> { order(id: :desc) }, class_name: "Comment"
  end

  Post.preload(:latest_comment).limit(5).map do |post|
    puts post.latest_comment
  end
end

Run a standalone playground

If you want to run a playground without the need to install the gem apart or with a Gemfile, you can use bundler/inline and run the playground like this:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'active_record_playground_runner', "~> 0.1.0", github: "bhserna/active_record_playground_runner", branch: "main"
end

require "active_record_playground_runner"

ActiveRecordPlaygroundRunner::Playground.new("My playground name") do
  schema do
    create_table :posts do |t|
      t.string :title
      t.text :body
    end

    create_table :comments do |t|
      t.text :body
      t.integer :post_id
    end

    add_index :comments, :post_id
  end

  models do
    class Post < ActiveRecord::Base
      has_many :comments
    end

    class Comment < ActiveRecord::Base
      belongs_to :post
    end
  end

  seeds do
    posts = create_list(Post, count: 10) do
      {
        title: FFaker::CheesyLingo.title,
        body: FFaker::CheesyLingo.paragraph
      }
    end

    create_list_for_each_record(Comment, records: posts, count: 20) do |post|
      {
        post_id: post.id,
        body: FFaker::CheesyLingo.sentence
      }
    end
  end

  example "first" do
    Post.includes(:comments).limit(5).map do |post|
      puts post.comments.count
    end
  end

  example "second" do
    Post.includes(:comments).limit(5).map do |post|
      puts post.comments.size
    end
  end
end.setup.run.destroy

And then run the file like any other ruby file:

ruby my_playground.rb

Mistakes we make with counts in rails

2022-06-06T22:51:00Z

Display the count of an association could look like a simple task, but in some cases it can give you real troubles.

It is very easy to end with n+1 queries. Counting with ruby sometimes can be ok, but sometimes could be impossible. Counting via sql is usually ok, but sometimes could be slow or maybe unnecessary.

Here I will share five mistakes with counts I have made and seen. It could help you to at least be aware of them.

1. Using slow sql counts in views

Sometimes counting via SQL using count or size to get a single value or by preloading counts can be slow depending of the number of records you are counting.

For example, imagine you need to put the number of likes in a post and you could have posts with thousands of likes. It could make your page slow.

# Somtimes an SQL count can make your
# page slow when the count is to large
post = Post.find(1)
post.likes.count #...

# Sometimes one count is ok but
# preloading a group of counts can be slow
posts = Post.limit(5)
likes = Like.where(post_id: posts).group(:post_id).count

posts.each do |post|
  puts "Post: #{post.id}, likes: #{likes[post.id] || 0}"
end

In those cases you can try a counter cache:

# You can use a counter cache
class Like < ApplicationRecord
  belongs_to :post, counter_cache: :true
end

class Post < ApplicationRecord
  has_many :likes
end

post = Post.find(1)
post.likes.size #... No query here

Or maybe fragment caching:

<p><%= post.abstract %></p>
<span>Likes: <%= posts.likes.count %></span>

<%= render partial: 'posts/post', collection: posts, cached: true %>

2. Count via sql an association you will use later

Sometimes in a page we show a count calculated via SQL using count or size, and later in the page we load the full collection we have just counted to display the information.

Something like this:

posts = Post.all

# Count post via sql (1 query)
puts posts.count

# Load posts (other query)
posts.each do |post|
  puts post.title
  puts post.body
end

In cases like this, you can load and count the collection with length or load.size to avoid making two calls to the database:

posts = Post.all

# Load posts and count with ruby (1 query)
puts posts.load.size

# Posts already loaded (no more queries)
posts.each do |post|
  puts post.title
  puts post.body
end

3. N+1 queries for counting associations inside a list

I think that this is the most common problem, asking the database for a count inside an each loop.

posts = Post.all

# Load posts (1 query)
posts.each do |post|
  # Count comments via sql (1 query per post)
  puts post.comments.count
end

To solve it you can try one of these three options:

4. Preload associations for counting inside a list

When you need to put a count inside an each loop and avoid n+1 sometimes we first preload the association and then count the elements in ruby.

posts = Post.preload(:comments)

# Load posts and comments (2 queries)
posts.each do |post|
  # Count with ruby (no queries)
  puts post.comments.size
end

This could be good enough in a lot of cases, but if the association have too many records and specially if you are not paginating the list, it could cause some performance and memory problems.

To solve it you can also try one of these three options:

5. Forget fragment caching

If you can enable caching in your project you can also try to avoid the n+1 queries or slow counts in the list with fragment caching.

You can think of n+1 as a feature, let the n+1 queries run just the first time and use the cache after that.

For example if you have a posts/_post partial like this:

<p><%= post.abstract %></p>
<span>Likes: <%= posts.likes.count %></span>

And then in the index view call the partial with something like this:

<%= render partial: 'posts/post', collection: posts, cached: true %>

You will have n+1 queries on the first render, but after that rails will read the cache.

You can go to the Caching with Rails: An Overview to learn more about rails caching.

A tool and examples to help you implement the first or last per record in rails with one query

2022-05-31T11:58:00Z

Have you ever needed to get the most recent post for each user in rails, but didn’t know how to do it without using map?

Or maybe something similar, like:

The first or last comment for each post
The first or last payment for each customer
The first or last review for each customer

Here is an example of a simple way to do it when you can use the id column to sort the records.

class Post < ActiveRecord::Base
  belongs_to :user

  scope :first_per_user, -> {
    where(id: select("min(id)").group(:user_id))
  }

  scope :last_per_user, -> {
    where(id: select("max(id)").group(:user_id))
  }
end

puts Post.first_per_user
# => SELECT "posts".*
#    FROM "posts"
#    WHERE "posts"."id"
#    IN (SELECT min(id) FROM "posts" GROUP BY "posts"."user_id")

puts Post.last_per_user
# => SELECT "posts".*
#    FROM "posts"
#    WHERE "posts"."id"
#    IN (SELECT max(id) FROM "posts" GROUP BY "posts"."user_id")

But here you will find a repo, with:

5 different methods to implement the “first or last per user”.
2 benchmarks, memory and ips, that you can run to test which method is better for your use case.
2 examples on how to associate the posts to the users, and do something like user.last_post or posts[user].

You can find the repo in: bhserna/first_or_last_per_user.

The 5 methods are:

The 2 benchamarks are:

The 2 examples on how to associates the posts to the users are:

How to run the examples

Install the dependencies with bundle install.
Database setup - run the command:

ruby db/setup.rb

Run the examples with ruby examples/<file name>. For example:

ruby example/00_example.rb

Change the seeds on db/seeds.rb and re-run ruby db/setup.rb to test different scenarios.

Active Record Playground

This repo uses the Active Record Playground that you can also use to play with some Active Record models in isolation and without creating a full rails app.

Inspiration

This example is based on a proposal of Steave Polito, but trying to implement the samething with one query.

Solve n+1 queries and slow counts with counter caches

2022-05-19T11:45:00Z

Sometimes calculating the count of a collection on every page render, even in SQL, is not the best option.

Imagine that you need to render a post, with its count of likes and comments, but the post has thousands of likes and comments. That would be slow.

One solution is to save the count of those comments and likes in the record, and when you need to render the post, instead of counting the elements again, just display the count that you already saved.

Keeping that count in sync is not easy, but rails makes it easy for us with a feature called “counter cache”.

Here I will show you how you can use a counter caches, how to introduce it in an existent association and some tips to work with them.

Add a counter cache to a new association with the rails defaults

To add a counter cache you will need to:

Add the counter_cache: true option in the belongs_to declaration.
Add a column to keep the count in the associated model (the one with the has_many), with the name #{table_name}_count.

For example to keep the count of likes in a post, you can add a likes_count coulumn in the posts table.

class Post < ApplicationRecord
  has_many :likes
end

class Like < ApplicationRecord
  # 1. Add the counter_cache: true option in the belongs_to declaration
  belongs_to :post, counter_cache: true
end

class CreatePosts < ActiveRecord::Migration[7.0]
  def change
    create_table :posts do |t|
      #...
      # 2. Add a column to keep the count in the associated model (has_many)
      t.integer :likes_count
    end

    create_table :likes do |t|
      t.references :post
    end
  end
end

Custom counter cache columns names

If you want to add a different name to the counter cache column you have to specify the name of the column in both association declarations, the belongs_to and the has_many.

class Post < ApplicationRecord
  # Specify the custom column name in the has_many association
  has_many :likes, counter_cache: :likes_total
end

class Like < ApplicationRecord
  # Specify the custom column name in the belongs_to association
  belongs_to :post, counter_cache: :likes_total
end

class CreatePosts < ActiveRecord::Migration[7.0]
  def change
    create_table :posts do |t|
      #...
      # 2. Add a column to keep the count in the associated model (has_many)
      t.integer :likes_total
    end

    create_table :likes do |t|
      t.references :post
    end
  end
end

It is important to remember to also define the counter_cache option in the has_many association. Without it, the size method won’t be able to read your counter cache column.

Reset the counters for an existent association with just a few records

If you want to add a counter cache to an existing association that already has records created, you will need to populate the counter column.

If you don’t have too many records to update you can use the reset_counters class method.

You can iterate through each record and reset its counter.

class Post < ApplicationRecord
  has_many :likes
end

class Like < ApplicationRecord
  # 1. Add the counter_cache: true option in the belongs_to declaration
  belongs_to :post, counter_cache: true
end

class AddLikesCountToPosts < ActiveRecord::Migration[7.0]
  def up
    # 2. Add a column to keep the count in the associated model (has_many)
    add_column :posts, :likes_count, :integer

    # 3. Update the counters for each post
    Post.find_each do |post|
      Post.reset_counters(post.id, :likes, touch: true) # The touch is optional
    end
  end

  def down
    remove_column :posts, :likes_count, :integer
  end
end

The reset_counters method will make a database call for each post, and if you have many records it will take a lot of time. Remember to just use this method if you don’t have too many records.

Reset the counters with SQL, for associations with many records

As I said before, If you will add the counter cache to an existing association that already has records, you will need to populate the counter column.

But If you have too many records to update, the previous method can be really slow, because it will make a database call for each record.

Other way of accomplish the task is by executing sql directly, like it is described by Ryan McGeary in the post Proper Counter Cache Migrations in Rails.

class Post < ApplicationRecord
  has_many :likes
end

class Like < ApplicationRecord
  # 1. Add the counter_cache: true option in the belongs_to declaration
  belongs_to :post, counter_cache: true
end

class AddLikesCountToPosts < ActiveRecord::Migration[7.0]
  def up
    # 2. Add a column to keep the count in the associated model (has_many)
    add_column :posts, :likes_count, :integer

    # 3. Update the counters for each post
    update_counters
  end

  def down
    remove_column :posts, :likes_count, :integer
  end

  def update_counters 
    execute <<-SQL.squish
      UPDATE posts
      SET likes_count = (
        SELECT count(1)
        FROM likes
        WHERE likes.post_id = posts.id
      )
    SQL
  end
end

How to use the counter cache in your views?

You can use it in two ways:

Using the method with the column name
Using the size method

class Post < ApplicationRecord
  has_many :likes
end

class Like < ApplicationRecord
  belongs_to :post, counter_cache: true
end

posts = Post.limit(10)
posts.each do |post|
  # Using the method with the column name
  post.likes_count

  # "size" will also read you counter cache column
  post.likes.size
end

If you are using a custom column name, you can also use both, the column name and the size method.

class Post < ApplicationRecord
  has_many :likes, counter_cache: :total_likes
end

class Like < ApplicationRecord
  belongs_to :post, counter_cache: :total_likes
end

posts = Post.limit(10)
posts.each do |post|
  # Using the method with the column name
  post.total_likes

  # "size" will also read you counter cache column
  post.likes.size
end

But, when you are using a custom counter cache name, is important to remember to also define the counter_cache option in the has_many association. Without it size won’t be able to read your counter cache column.

class Post < ApplicationRecord
  # If you don't define the custom counter cache column...
  has_many :likes
end

class Like < ApplicationRecord
  belongs_to :post, counter_cache: :total_likes
end

posts = Post.limit(10)
posts.each do |post|
  # "size" won't know the custom column name and will execute a COUNT query
  post.likes.size
end

References

4 things you can try before using a counter cache in rails

2022-05-16T22:18:00Z

Maybe you have heard about the counter cache feature.

A counter cache makes finding the number of belonging objects more efficient by keeping a column with the count.

Rails makes it easy to implement it, but is not free. Sometimes it may be better not to use it.

Here are four things you could try before or instead of using a counter cache, without introuducing n+1 queries.

1. If you don’t need the count in a list just count the records

If you don’t need to use the count inside a list, and instead you just want to show the count of the association in a details or stats page, you can try to count the records via SQL using count or size.

post = Post.first
# Post Load ...
post.comments.size
# Comment Count (0.4ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

# or...
post.comments.count
# Comment Count (0.4ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

If you need to load all the records in the association in the same page and without pagination, you can load the records and count the records with ruby using length or load.size, to make just one call to the database instead of two.

post = Post.first
# Post Load ...

# Length will load the coments and count the elements in ruby
# avoiding one extra database call.
post.comments.length
# Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

# or...
# #load will load the coments and #size will count the elements
# in ruby avoiding one extra database call.
post.comments.load.size
# Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

If you have doubt when is better to use count vs size vs length you can read the post: Difference between count, length and size in an association with ActiveRecord.

2. Preload the records in a list

If you need to display the count of an association for each record in a list, and you know the records wont’t have too many associated records, you can try to preload the records and use lenght or size to count the records in ruby.

# Preload the full association
posts.preload(:comments).each do |post|

  # Use size to count with ruby
  post.comments.size

  # or lenght
  post.comments.length
end

But be carefull, because if the association has too many records, you can introduce a performance problem.

3. Preload the counts you’ll need in a list

If you need to display the count of an association for each record in a list, and you know the records will have too many associated records, as I said before, preloading all the associated records can introduce a performance problem.

Instead you can try to preload just the counts with something like this:

posts = Post.limit(5)
# Filter the likes for those posts and group them by count
likes = Like.where(post_id: posts).group(:post_id).count

posts.each do |post|
  # Find the likes count for the current post
  # and fallback to zero if no count
  puts "Post: #{post.id}, likes: #{likes[post.id] || 0}"
end

If you want to see other ways to preload the counts, you can see the post: How to preload counts in a list with ActiveRecord.

4. Fragment caching

If you can enable caching in your project you can also try to avoid the n+1 queries in the list with fragment caching.

You can think of n+1 as a feature, and let the n+1 queries run just the first time and use the cache after that.

For example if you have a posts/_post partial like this:

<p><%= post.abstract %></p>
<span>Likes: <%= posts.likes.size %></span>

And then in the index view, you call the partial with something like this:

<%= render partial: 'posts/post', collection: posts, cached: true %>

You will have n+1 queries on the first render, but after that, rails will read the cache instead avoiding the n+1 queries.

To learn more about rails caching you can go to: Caching with Rails: An Overview

Warning: Don’t try too hard to avoid the counter cache

Although sometimes you can avoid the counter cache with these tactics, sometimes these tactics can also give you some troubles.

So take this ideas as suggestions, but don’t think that one of this will solve all your problems every time.

Take them as options to help you solve your current problem.

When do you need a counter cache in rails?

2022-05-09T22:12:00Z

Maybe you have heard about the counter cache feature.

It makes finding the number of belonging objects more efficient, by keeping a column with the count.

But… When should you use it? Do you need to have a counter cache for every association count in your app?

Here are three situations when you could need a counter cache.

When you have slow counts in your views

Even if you do the count via SQL, a single count can be slow enough to need a counter cache.

Imagine you need to put the number of likes in a post.

If a post could have thousands or millions of likes, a single count, can really hit the performance of the app.

posts = Post.find(id)

# If there are thousands or millions of likes
# counting via SQL could also be slow
puts "Post likes: #{posts.likes.size}"

In this case you could try a counter cache or maybe fragment caching.

When preloading the counts in a list is slow

Sometimes preloading counts for a list via SQL is not fast enough.

Imagine you need to put the number of likes for each post in a list.

If there could be posts with thousands or millions of likes, the count for a list of posts could be really slow.

Even if you are already just preloading the counts to avoid n+1 queries with something like this:

posts = Post.limit(5)

# If there are thousands or millions of likes per post
# preloading the counts could also be slow
likes = Like.where(post_id: posts).group(:post_id).count

posts.each do |post|
  puts "Post: #{post.id}, likes: #{likes[post.id] || 0}"
end

In this case you could also try a counter cache or maybe fragment caching.

When you use a lot a count in the application

Sometimes you will be using the count in a lot of places in the app.

You will need to preload the count in every place, raising the possibility to introduce n+1 queries.

In this situation it could be a good option to implement a counter cache, even if you already have a preload object.

Use dom_id helper on capybara with rspec and rails

2022-05-02T22:14:00Z

At least for me it is very common to use the within method in capybara, to match the dom_id output.

And is very common to do it with code like "#user_#{user.id}" or #project_#{project.id}.

In this way:

within "#user_#{user.id}" do
  expect(page).to have_text user.name
end

# or...

within "#project_#{project.id}" do
  expect(page).to have_text project.name
end

If you do it one time, there is no trouble, but if you start doing it in every file, maybe you will want to do something to remove the repetition.

Here I will show you two tips, to help you avoid this repetition.

Tip 1: Include ActionView::RecordIdentifier module to use the dom_id on your specs

On your spec you can include ActionView::RecordIdentifier and use the dom_id helper in your spec/scenario.

RSpec.describe "My spec" do
  include ActionView::RecordIdentifier

  scenario do
    within "##{dom_id(user)}" do
      expect(page).to have_text user.name
    end

    within "##{dom_id(project)}" do
      expect(page).to have_text project.name
    end
  end
end

You can do it because the module ActionView::RecordIdentifier defines the method dom_id.

If you want to use you the helper in all your specs you can include the module inside you rspec configuration block like this:

RSpec.configure do |config|
  config.include ActionView::RecordIdentifier, type: :system
end

And now you will be able to use the helper without including the module on each spec file.

RSpec.describe "My spec" do
  scenario do
    within "##{dom_id(user)}" do
      expect(page).to have_text user.name
    end

    within "##{dom_id(project)}" do
      expect(page).to have_text project.name
    end
  end
end

Tip 2: Create a css_id helper to also avoid the interpolation

If you don’t want to repeat yourself doing "##{dom_id(user)}" all over the place, you can write a method css_id (or whatever you want to call it) to do it for you.

You can add a new file spec/support/record_identifier_helper.rb and write something like this:

module RecordIdentifierHelper
  include ActionView::RecordIdentifier

  def css_id(*args)
    "#" + dom_id(*args)
  end
end

Then you need to be sure that in your rails_helper you are loading that file.

# Normally you will have something like this.
Dir[File.join(__dir__, "system/support/**/*.rb")].sort.each { |file| require file }

Then include your module on your rspec configuration block:

RSpec.configure do |config|
  config.include RecordIdentifierHelper, type: :system
end

And now you will be able to use the helper on your specs, like this:

RSpec.describe "My spec" do
  scenario do
    within css_id(user) do
      expect(page).to have_text user.name
    end

    within css_id(project) do
      expect(page).to have_text project.name
    end
  end
end

How to preload counts in a list with ActiveRecord

2022-04-25T22:19:00Z

Imagine that you need to put the number of likes for each post in a list, but avoiding n+1 queries.

posts.each do |post|
  post.likes.count # n+1 queries
end

One way to avoid n+1 queries here, is to preload the association and the count the records in ruby. But if the association has too many records, you could introduce a performance problem.

posts.preload(:likes).each do |post|
  post.likes.length
end

To solve it you can try to preload just the counts, here are four ways of doing it (and a bonus).

Method 1: Count from the associated model

You will need to…

Filter the associated models for the records on your list
Group and count by the foreign_key to obtain a hash with the form { id ⇒ count }
Use that hash on the list to get the count for each record
Fallback to zero if no count

Here the example with posts and likes:

# Given a list of posts
posts = Post.limit(5)

# 1. Filter the likes for those posts
likes = Like.where(post_id: posts)

# 2. Group by :post_id and count
counts = likes.group(:post_id).count

puts counts

posts.each do |post|
  # 3. Find the likes count for the current post
  # 4. Fallback to zero if no count
  puts "Post: #{post.id}, likes: #{counts[post.id] || 0}"
end

# Like Count (0.8ms)  SELECT COUNT(*) AS "count_all", "likes"."post_id" AS "likes_post_id" FROM "likes" WHERE "likes"."post_id" IN (SELECT "posts"."id" FROM "posts" LIMIT $1) GROUP BY "likes"."post_id"  [["LIMIT", 5]]
# {3=>20, 2=>20, 4=>20}
# Post Load (0.2ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
# Post: 1, likes: 0
# Post: 2, likes: 20
# Post: 3, likes: 20
# Post: 4, likes: 20
# Post: 5, likes: 0

Method 2: Count joining the associated model

You will need to…

Join the association
Group by the id of the base table and count the ids of the associated records to obtain a hash with the form { id ⇒ count }
Use that hash on the list to get the count for each record
Fallback to zero if no count

Here the example with posts and likes:

# Given a list of posts
posts = Post.limit(5)

# 1. Join the likes
# 2. Group by "posts.id" and count "likes.id"
counts = posts.joins(:likes).group("posts.id").count("likes.id")

puts counts

posts.each do |post|
  # 3. Find the likes count for the current post
  # 4. Fallback to zero if no count
  puts "Post: #{post.id}, likes: #{counts[post.id] || 0}"
end

# Post Count (1.0ms)  SELECT COUNT("likes"."id") AS "count_likes_id", "posts"."id" AS "posts_id" FROM "posts" INNER JOIN "likes" ON "likes"."post_id" = "posts"."id" GROUP BY "posts"."id" LIMIT $1  [["LIMIT", 5]]
# {2=>20, 3=>20, 4=>20, 8=>20}
# Post Load (0.3ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
# Post: 1, likes: 0
# Post: 2, likes: 20
# Post: 3, likes: 20
# Post: 4, likes: 20
# Post: 5, likes: 0

Method 3: Count left joining the associated model

You will need to…

Left join the association
Group by the id of the base table and count the ids of the associated records to obtain a hash with the form { id ⇒ count }
Use that hash on the list to get the count for each record

You don’t need a fallback to zero, because the left join will return a count for all posts.

Here the example with posts and likes:

# Given a list of posts
posts = Post.limit(5)

# 1. Left join the likes
# 2. Group by "posts.id" and count "likes.id"
counts = posts.left_joins(:likes).group("posts.id").count("likes.id")

puts counts

posts.each do |post|
  # 3. Find the likes count for the current post
  # You don't need fallback
  puts "Post: #{post.id}, likes: #{counts[post.id]}"
end

# Post Count (0.4ms)  SELECT COUNT("likes"."id") AS "count_likes_id", "posts"."id" AS "posts_id" FROM "posts" LEFT OUTER JOIN "likes" ON "likes"."post_id" = "posts"."id" GROUP BY "posts"."id" LIMIT $1  [["LIMIT", 5]]
# {1=>0, 2=>20, 3=>20, 4=>20, 5=>0}
# Post Load (0.2ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
# Post: 1, likes: 0
# Post: 2, likes: 20
# Post: 3, likes: 20
# Post: 4, likes: 20
# Post: 5, likes: 0

Method 4: Add the count to the selected values

You will need to…

Left join the association
Select the records table and the associations count
Group by the id of the base table
Use the associations count as a method in the records (Ej. likes_count)

You don’t need a fallback to zero, because the left join will return a count for all posts.

Here the example with posts and likes:

# Given a list of posts
posts = Post.limit(5)

# 1. Join the likes
# 2. Select the posts and the likes count
# 3. Group by posts.id
posts = posts
  .left_joins(:likes)
  .select("posts.*, COUNT(likes.id) AS likes_count")
  .group("posts.id")

posts.each do |post|
  # 4. Use post.likes_count
  # You don't need fallback
  puts "Post: #{post.id}, likes: #{post.likes_count}"
end

# Post Load (1.0ms)  SELECT posts.*, COUNT(likes.id) AS likes_count FROM "posts" LEFT OUTER JOIN "likes" ON "likes"."post_id" = "posts"."id" GROUP BY "posts"."id" LIMIT $1  [["LIMIT", 5]]
# Post: 1, likes: 0
# Post: 2, likes: 20
# Post: 3, likes: 20
# Post: 4, likes: 20
# Post: 5, likes: 0

Bonus: Use a preload object with one of the first three methods

If you don’t want to have this code in the view or controller, you can write a preload object to do it. It could be something like this:

class LikesCounts
  attr_reader :posts

  def initialize(posts)
    @posts = posts
  end

  def [](post)
    # 3. Find the likes count for the current post
    # 4. Fallback to zero if no count
    counts[post.id] || 0
  end

  def counts
    # 1. Filter the likes for those posts
    # 2. Group by :post_id and count
    @counts ||= Like.where(post_id: posts).group(:post_id).count
  end
end

posts = Post.limit(5)
counts = LikesCounts.new(posts)

posts.each do |post|
  puts "Post: #{post.id}, likes: #{counts[post]}"
end

# Post Load (0.7ms)  SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
# Like Count (0.4ms)  SELECT COUNT(*) AS "count_all", "likes"."post_id" AS "likes_post_id" FROM "likes" WHERE "likes"."post_id" IN (SELECT "posts"."id" FROM "posts" LIMIT $1) GROUP BY "likes"."post_id"  [["LIMIT", 5]]
# Post: 1, likes: 0
# Post: 2, likes: 20
# Post: 3, likes: 20
# Post: 4, likes: 20
# Post: 5, likes: 0

Difference between count, length and size in an association with ActiveRecord

2022-04-18T22:11:00Z

With Rails/ActiveRecord, you can count the records in an association using three methods .count, .length, .size.

As a rails newbie, when you first discover this, it can be confusing… Why do we need three methods? Is there any advantage to using one method over the other? How are they different?

Here are 6 characteristics of this three methods that you should know. They will help you decide which method to use on different situations.

I will also share a why I think that most of the time you should start with .size and the reference to the docs where you can find more information.

1. “count” always performs an SQL `COUNT` query

Even if you call it twice.

post = Post.first
# Post Load ...
post.comments.count
# Comment Count (0.6ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.count
# Comment Count (0.3ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

Even if the collection has already been loaded it will always execute a COUNT

post = Post.first
# Post Load ...
post.comments.load
# Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.count
# Comment Count (0.5ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

2. “length” loads the records and calculates the length with ruby

If you later need to use the records, they will be already loaded.

post = Post.first
# Post Load ...
post.comments.length
# Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.to_a
# (no query)

3. “length” if loaded, only calculates the length with ruby

If the collection is already loaded, it won’t need a new database call.

post = Post.first
# Post Load ...
post.comments.to_a
# Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.length
# (no query)

4. “size” if loaded, calculates the length with ruby

Like with length, if the collection is already loaded it will compute the length with ruby.

post = Post.first
# Post Load ...
post.comments.load
# Comment Load (0.5ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.size
# (no query)

5. “size” if not loaded, performs an SQL `COUNT` query

Like count, it will execute a COUNT if the collection has not been loaded.

post = Post.first
# Post Load ...
post.comments.size
# Comment Count (0.4ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.load
# Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

And also like count, if you use it twice it will make two queries.

post = Post.first
# Post Load ...
post.comments.size
# Comment Count (1.5ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]
post.comments.size
# Comment Count (0.7ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = $1  [["post_id", 1]]

6. “size” if there is a counter cache, uses the cached count

It will never need a database call.

post = Post.first
# Post Load ...
post.likes.size
# (no query)
post.likes.size
# (no query)

Conclusion: Start with “size”

Most people recommend to use .size unless you have a reason for not using it, because most of the time it will do what you want:

It will use the counter cache if is there
It will count with ruby if the records are already loaded
It will perform an SQL COUNT if the records are not already loaded

When you don’t have a counter cache and you need to use the association later in the view, one tip from the Nate Berkopec, is to use .load.size.

Reference

Active record playground, for rails developers that want an easy way to practice with active record in isolation

2022-03-29T12:21:00Z

I want to share with you my “Active record playground template”.

It is a template that you could use as a guide or by cloning the repo and following the instructions in the README if you want to play with some Active Record models in isolation and without creating a full rails app.

I was repeating my self over and over

I was setting up the structure of this template from scratch over and over again until I decided to do it in a little more general way.

Now it is much more faster and easier for me to start playing with Active Record in isolation. And now, I hope it also can help you.

The default structure of this template will make some decision for you:

1. The starting Gemfile

It includes activerecord and pg, but also the faker gem, because it will be help you to define your examples data.

2. A place for your models

On lib/models.rb you can define the objects/records that you need. This file is already required to run your seeds and examples.

3. A place for your seeds

On db/seed.rb you will be able to define your seeds using the objects defined on lib/models.rb. You will also be able to run them with “ruby db/seed.rb”.

4. A place for the schema

As this is “playing” code, instead of migration, you will create directly the schema on db/schema.rb.

5. The db setup command

With “ruby db/setup.rb” you will create or re-create the database, load the schema and run the seeds.

6. A place to define your examples

You will have an /examples dir, to add the examples files that you need.

Requiring examples/config will require the database initialization code and the models defined on lib/models.rb

You can run the examples with ruby examples/<file name>

ruby example/00_example.rb

Use the template

Use this template if you want to play with Active Record in isolation without creating a full rails app.

You can find the template on https://github.com/bhserna/active_record_playground

You can use it as a guide or by cloning the repo and following the instructions on the README.

Some examples of using this template

Here are some examples that I have shared, using this template (or previous versions of it):

Specific preloading with scoped associations

2022-03-16T12:44:00Z

It is very common the need to preload an association then filter that association and then use the filtered collection.

For example, imagine that you need to render a list of posts with just its “popular” comments, where “popular” means “with a likes_count greater than or equals to X”.

Approach without scoped association

So you can define Comment#popular? like this…

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  POPULAR = 9

  belongs_to :post

  def popular?
    likes_count >= POPULAR
  end
end

And so you can sometimes, complete your task, avoiding n+1 queries, by preloading the comments and then selecting just the “popular” comments with ruby.

Something like this…

# Preload all comments for each post
posts = Post.preload(:comments).limit(5)
# => SELECT "posts".* FROM "posts" LIMIT $1
# => SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5)

posts.each do |post|
  render_title(post)

  # Select just the popular comments
  post.comments.select(&:popular?).each do |comment|
    render_comment(comment)
  end
end

Possible memory problems

This can work on many situations, but sometimes it can give you some troubles, with memory, if the association that you are preloading has a lot of records, because when you preload an association rails will instantiate all the associated records.

Solution with scoped association

One way of solving this problem with rails is to create an association passing an scope, to filter the associated records.

For example, in this problem, you can add a popular_comments associations, to help you preload just the comments that you are going to need.

Like this…

class Post < ActiveRecord::Base
  has_many :comments

  # Here you are defining an association, that will use Comment.popular,
  # to filter the Comment records.
  has_many :popular_comments, -> { popular }, class_name: "Comment"
end

class Comment < ActiveRecord::Base
  POPULAR = 9

  belongs_to :post

  # Here we are definining what `popular` means.
  scope :popular, -> { where(likes_count: POPULAR..) }
end

And then use the popular_comments association to preload just the “popular” comments instead of all the comments for each post, and then use the popular_comments method to display just the “popular” comments.

# Preload directly just the popular comments
posts = Post.preload(:popular_comments).limit(5)
# => SELECT "posts".* FROM "posts" LIMIT $1  [["LIMIT", 5]]
# => SELECT "comments".* FROM "comments" WHERE "comments"."likes_count" >= $1 AND "comments"."post_id" IN ($1, $2, $3, $4, $5)

posts.each do |post|
  render_title(post)

  # You can call the association for each post without n+1 queries
  # because the association has already been preloaded
  post.popular_comments.each do |comment|
    render_comment(comment)
  end
end

In this way you can use your defined scopes, to build “scoped” associations to help you preload just the records that your are going to need and save some memory.

Run the examples

You can find a repo with the examples from this posts, on github.com/bhserna/specific_preloading_with_scoped_associations.

How to use react with hotwire?

2022-02-28T23:14:00Z

I have seen that some people think that you have to forget about Rails and Hotwire, if you want to use React or another Javascript framework to build a Single Page Aplication, but this idea is not true.

You can use Rails and Hotwire as the base of your app, and plug your React app to be shown where is needed. In this way you can use both frameworks, and pick the one that you think that will help you must for each feature.

Here I want to explain how you can plug a react app inside a Rails app with Hotwire, using Stimulus.js.

You will need a javascript bundler

To use a javascript bundler with rails, you can use the gem jsbundling-rails, it will help you to integrate esbuild, rollup.js, or Webpack to bundle your JavaScript, then deliver it via the asset pipeline in Rails.

If you are already familiar with one of those, use it, if you are not familiar, or you want to move away from webpack, I think that you could try esbuild.

I will continue this article explaining who you can integrate react using esbuild.

Install jsbundler and esbuild tool

Following the instructions from the github page, you will need to:

Add jsbundling-rails to your Gemfile with gem 'jsbundling-rails'
Run ./bin/bundle install
Run ./bin/rails javascript:install:esbuild

After this steps the installer should have updated your package.json adding esbuild as dependency and a build script, with the instruction to build your javascript from app/javascript and put it on app/assets/builds.

{
  "name": "app",
   "private": "true",
   "dependencies": {
     "@hotwired/stimulus": "^3.0.1",
-    "@hotwired/turbo-rails": "^7.1.1"
+    "@hotwired/turbo-rails": "^7.1.1",
+    "esbuild": "^0.14.18"
+  },
+  "scripts": {
+    "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds"
+  }
 }

Install react and react dom

Now you can go and install react with yarn:

yarn add react react-dom

It will update you package.json with react and react-dom.

{
  "name": "app",
   "private": "true",
   "dependencies": {
     "@hotwired/stimulus": "^3.0.1",
     "@hotwired/turbo-rails": "^7.1.1",
-    "esbuild": "^0.14.18"
+    "esbuild": "^0.14.18",
+    "react": "^17.0.2",
+    "react-dom": "^17.0.2"
   },
   "scripts": {
     "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds"
   }
 }

The hello world

You can now add a app/javasctipts/App.jsx file to test the react integration. There you can add something like:

import * as React from 'react'

export default () => <h1>Hello, world!</h1>

And now add a app/javascript/controllers/hello_controller.jsx that to display the App component on connect .

Note: 👉 If you already have one hello_controller.jsx, because you are on a new rails app, you can just change the file extension to .jsx.

import { Controller } from "@hotwired/stimulus"
import * as React from 'react'
import * as ReactDOM from "react-dom";
import App from "./../app"

export default class extends Controller {
  connect() {
    ReactDOM.render(<App />, this.element);
  }
}

And now on a page add a tag for the hello_controller to display the react App component.

<div data-controller="hello">
</div>

At this moment you should see “Hello world!” on the page if you visit the page.

By starting the react app with Stimulus.js we can then move to other pages, that don’t have any react component, and then go back to the page with react using turbo, and Stimulus.js will initialize the react app as expected.

Adding more than one file

You can add more jsx files, directly on app/javascripts.

For example if you add a Counter.jsx with:

import * as React from 'react';
import { useState } from 'react';

export default function Counter() {
  const [count, setCount] = useState(0);

  function handleClick() {
    setCount(count + 1);
  }

  return (
    <button onClick={handleClick}>
      Clicked {count} times
    </button>
  );
}

You can then import it on the App.jsx with…

import * as React from 'react'
import Counter from "./Counter";

export default () => <h1>Hello, world!</h1>

And with that you can now can use that Counter.

For example, we can change the App function to something like this…

import * as React from 'react'
import Counter from "./Counter";

export default function App() {
  let style = {
    border: "1px solid red",
    padding: "1rem"
  }

  return (
    <div style={style}>
      <h1>This is a React App!</h1>
      <p>This are some react counters!</p>

      <Counter />
      <Counter />
      <Counter />
    </div>
  );
}

To show something like this…

Example app

I built a little example app with the last example.

You can get the code of the example app on github.com/bhserna/hotwire-react.

It is a demo that shows how you can plug a react app inside a rails app using Stimulus.js.

If you go to the “page with react” you will see a page with the counters implemented using react and started using a Stimulus controller.

As we saw in this post, by starting the react app with Stimulus.js you can then move back to the index, that don’t have any react component, and the go back to the page with react using turbo, and Stimulus.js will initialize the app as expected.

Like in this video…

Should you split your rails app into backend and frontend?

2022-02-14T23:23:00Z

When you start a new app or when things start to get complicated on the frontend side of your rails app, I think that it is normal to start thinking if you should split the app into a rails backend and a frontend with react, vue, or other javascript framework.

I am a rails developer that normally prefers the “majestic monolith”, and I will normally recommend to not doing it, and even I sometimes think if I should split my apps or not. But I think that at least in my this normally caused by hype or frustration.

So this is a list for you and for me, with some things to consider when you have “reasons” for splitting your app like…

Reason: UI becomes more complex and we need/want a smooth user experience

Maybe you are thinking that with rails every interaction will be server side rendering….

That is not really true, rails by default propose server side rendering, but if some parts of your app requires a more complex interaction, you can always use a javascript framework for those parts, and with stimulus.js you will be able to plug that behavior in a “rails way”.

The talk Turbolinks 5: I Can’t Believe It’s Not Native! by Sam Stephenson describes this idea back on the “Turbolinks 5” times, but now the environment is much better.

Reason: The UI requires complex interactions on many parts of the app

Are those parts the majority of the app? Do you want to increase the complexity of the whole app for does parts? Can you plug those apps with stimulus.js?

Reason: We already have Vue/JQuery/Javascript code all over the place and it’s a mess

Was the mess caused by not splitting? Could stimulus.js and turbo help to reduce that complexity?

After that mess… Do you now have and idea of how you can structure your code better?

Reason: It is easier to test an api

When our system tests start getting slow and complicated we can think that is easier to have two separate test suites, one for the api and one for the front end app.

And maybe it is true that is easier to test both things as a separate thing, but it is just easier as it is easier to write a test for a single method than a system test.

If you split are you going to avoid system tests that execute through both parts? Are you going to feel the same confidence?

Reason: The backend will be leaner (less gems etc) and hopefully faster

Maybe this could be true, but how much? What about the performance of the front end app? What about the complexity of the whole project? What about the whole set of dependencies including the javascript dependencies?

Reason: It’s super hard to find great full stack rails developers

This is true, but I think that there is no easy path, finding good engineers is not an easy job.

It will be hard to find a great full stack rails developer, but it will also be hard to find a great frontend developer and/or backend developer.

Conclusion

As you have seen I am very biased to the monolith, maybe this should not be your only reference for this decision, but I hope it could be useful in some way.

How the turbo_frame_tag helper transforms an object to an html id?

2022-02-08T23:24:00Z

If you are new with rails and hotwire, it could be useful to know that you can pass an object to the turbo_frame_tag and it will use a string representation of that object as an id.

For example if you have a Product record, you can do:

product = Product.find(1)
turbo_frame_tag(product)

And it will return:

<turbo-frame id="product_1"></turbo_frame>

And if you do

product = Product.new
turbo_frame_tag(product)

It will return:

<turbo-frame id="new_product"></turbo_frame>

This happen because internally turbo_frame_tag uses the dom_id helper.

def turbo_frame_tag(id, src: nil, target: nil, **attributes, &block)
  id = id.respond_to?(:to_key) ? dom_id(id) : id
  #...
end

You can see the code of the turbo_frame_tag helper here:

https://github.com/hotwired/turbo-rails/blob/v1.0.1/app/helpers/turbo/frames_helper.rb

And the docs for the dom_id helper here:

https://api.rubyonrails.org/classes/ActionView/RecordIdentifier.html

Styler and tachyons examples

2022-01-28T23:30:00Z

I built a tool to comples css classes like this…

styles = Styler.new do
  style :default, ["pa3", "white", "bg_blue"]
  style :danger, [default - "bg_blue", "bg_red"]
end

styles.default.to_s # => "pa3 white bg_blue"
styles.danger.to_s # => "pa3 white bg_red"

… because I think that, although css is awesome (really!), I think that it misses is the ability to compose already defined styles, to define new ones.

In the introduction post and on github, you can find a tutorial of how the tool works.

But until now there was no “real” examples of how to use it.

A collection of examples

That is why I am building a site with some examples from the list of tachyons components from the Tachyons’s page, to show you how you can use the the gem to define the styles.

You can visit the site on styler-tachyouns-examples.bhserna.com.

The code of the site

If you want to help to add more examples, or you want to see how I used the tool to style the site, or if you are curious about how it was build, you can see the code on github/bhserna/styler_tachyons_examples.

How do you think that this tool can help you?

I know that maybe it is a little unexpected way of doing things, but if you have have some ideas on how this tool can help you it would be nice to know them.

You can write a comment here or tell me via twitter.

Inline CRUD with rails and hotwire

2022-01-11T00:08:00Z

This is a guide to help you build “inline CRUDs” with rails and hotwire, where you create and edit records in the same page.

Instead of a “new product” page it shows the form in the same page
Instead of an “edit product” page it shows the form on the same page by replacing the product item.

Like in the next video…

The example app

You can find the code of the app in the video on: github/bhserna/inline_crud_hotwire

To explain how you can add this behavior to your app, I will use the app from the video as an example.

If you need help to extrapolate the example for your use case, you can ask a question in the comments section of this post, I will try to answer there.

Inline CRUD interactions

I will try to explain the relevant parts on the interaction with turbo that will help you build an “inline CRUD”.

Show the “new product” form in the same page
Show the product after it is created
Show the form errors on the “new product” form
Show the “edit product” form on the item’s place
Show the product after it is updated
Show the form errors on the “edit product” form
Delete the product

Show the “new product” form in the same page

Add a <%= turbo_frame_tag "new_product", target: "_top" %> on the index page.
Add data-turbo-frame="new_product" to the “New product” link.
On the new template wrap the form on a <%= turbo_frame_tag @product do %>...<% end %>

<%=# products/index.html.erb %>
<%= turbo_frame_tag "new_product", target: "_top" %>

<%=# products/index.html.erb %>
<%= link_to "New product", new_product_path, data: { turbo_frame: "new_product" } %>

<%=# products/new.html.erb %>
<%= turbo_frame_tag @product do %>
  <%= render "form", product: @product %>
<% end %>

The <%= turbo_frame_tag "new_product", target: "_top" %> on the index page will be converted to a <turbo-frame id="new_product" target="_top">. It will set the place for the “new product” form.

The id is new_product because it will help us to latter use the method turbo_frame_tag @product where @product is Product.new, then rails will turn this into the string "new_product".

The target="_top" is there to target the whole page by default and in that way when the product is successfully created you can redirect to the index page and turbo will update the whole page.

The data-turbo-frame="new_product" attribute on the “New product” link. Tells turbo to put the content of the matching turbo frame from the server response on the <turbo-frame id="new_product"> that is on the current page.

This is why on the new.html.erb template the form is inside <%= turbo_frame_tag @product do %>, that will be transformed by rails to <turbo-frame id="new_product">.

Show the product after it is created

Redirect to the index page.

If the turbo frame has the attribute target="_top" to target the whole page by default you can redirect to the index page on the controller and turbo will update the whole page.

def create
  @product = Product.new(product_params)

  if @product.save
    redirect_to products_path
  else
    #...
  end
end

Show the form errors on the “new product” form

Update the form with a turbo_stream.

If the turbo frame has the attribute target="_top" to target the whole page by default you can’t render new.html.erb because then turbo will update the whole page with the response.

That’s why you need to create a new template new.turbo_stream.erb with:

<%= turbo_stream.update @product do %>
  <%= render "form", product: @product %>
<% end %>

Then Rails will change the <%= turbo_stream.update @product do %> to <turbo-stream action="update" target="new_product"> because the @product was not saved and its dom id representation is still "new_product".

In the controller you can respond with render :new, status: :unprocessable_entity and as the request was a TURBO_STREAM request rails will use the turbo_stream template by default.

def create
  @product = Product.new(product_params)

  if @product.save
    #...
  else
    render :new, status: :unprocessable_entity
  end
end

Show the “edit product” form on the item’s place

Wrap the list item in a <%= turbo_frame_tag product %>
On the edit template wrap the form on a <%= turbo_frame_tag @product do %>...<% end %>

Wrapping the list item in a <%= turbo_frame_tag product %>, creates a <turbo-frame id="product_#{product.id}">.

<%= turbo_frame_tag product do %>
  <div class="product-card">
    ...
    <div class="product-actions">
      <%= link_to "Edit", edit_product_path(product) %>
      ...
    </div>
  </div>
<% end %>

This will tell turbo to capture all navigation within the frame by expecting any followed link or form submission to return a response including a matching frame tag.

Wrapping the product edition form, on a <%= turbo_frame_tag @product do %>...<% end %> will create the matching <turbo-frame id="product_#{product.id}">.

<%=# products/edit.html.erb %>
<%= turbo_frame_tag @product do %>
  <%= render "form", product: @product %>
<% end %>

Then turbo will take the content from the matching <turbo-frame> the response, and update the content of the current <turbo-frame> on the page.

Show the product after it is updated

Redirect to the index page.

As the form was inside the <turbo-frame id="product_#{product.id}"> and the index page has a matching <turbo-frame id="product_#{product.id}">, turbo will take that content and update the form with the regular list item from the index page.

def update
  @product = Product.find(params[:id])

  if @product.update(product_params)
    redirect_to products_path
  else
    #...
  end
end

Show the form errors on the “edit product” form

Render the edit.html.erb template with an error status.

When you submit a form, tubo will expect a redirect, but if you return an error status, turbo will process the response. So if you respond with render :edit, status: :unprocessable_entity turbo will take the content from the matching <turbo-frame id="product_#{product.id}">, and update the page.

def update
  @product = Product.find(params[:id])

  if @product.update(product_params)
    #...
  else
    render :edit, status: :unprocessable_entity
  end
end

Delete the product

Redirect to the index page.

As the delete button, that is really an html form, was inside the <turbo-frame id="product_#{product.id}"> and the index page (after the item was deleted), will not have a matching <turbo-frame id="product_#{product.id}">, turbo will remove the content from <turbo-frame>.

Note that it will not remove the turbo frame from the page, it will just remove its content. If you want to remove the turbo frame you will need to respond with a turbo-stream to remove the turbo-frame.

Remote modals with rails, hotwire and bootstrap

2022-01-04T13:08:00Z

Here I want to show you how you can build “remote modals” (modals where the content is requested to the server), submit forms embedded in them and handle errors, with rails, hotwire and bootstrap.

Example app

I will show you how to build is something like this…

Where you can…

Click a link that requests content to the server and then shown inside a modal.
Submit a form and display the errors inside the modal
If the form is processed with success, the redirect to the expected page.
Keep the “fade” effect of the bootstrap modal.

Like a standard rails CRUD, but using modals to show the form.

Play with the code

The code of the app is on github.com/bhserna/remote modals hotwire bootstrap, you can clone it and play with it.

Versions

For the dependencies that I will talk about here, I am using this versions:

Ruby: 3.0.0
Rails: 7.0.0
Turbo rails: 1.0.0
Stimulus rails: 1.0.2
Bootstrap: 5.1.3

Show a form in modal

In the example app you can see this action twice:

On the “New product” link
And on the “Edit” link

Let’s see how the “New product” link works…

If you go to views/products/index.html.haml, you will see that the link has a data-turbo-frame="remote_modal" attribute.

%div= link_to "New product", new_product_path, class: "btn btn-primary",
  data: { turbo_frame: "remote_modal" }

This will tell turbo, to look for an already defined turbo_frame_tag with the id remote_modal.

This tag is defined on views/layouts/application.html.erb

<%= turbo_frame_tag "remote_modal", target: "_top" %>

With that tag defined, turbo will expect a server response with a matching turbo_frame_tag.

That means that a <turbo-frame id="remote_modal"> element should be somewhere in the server response.

This is implemented like this…

If you see the products controller you will see that it just sets the @product instance variable and renders the default template new.html.

def new
  @product = Product.new 
end

On view/products/new.html.haml you will see that it renders the "remote_modal" partial.

= render "remote_modal", title: "New product" do
  = render "form", product: @product

This partial is on views/application/_remote_modal.html.haml…

= turbo_frame_tag "remote_modal" do
  .modal.fade(tabindex="-1" data-controller="remote-modal" data-action="turbo:before-render@document->remote-modal#hideBeforeRender")
    .modal-dialog
      .modal-content
        .modal-header
          %h5.modal-title= title
          %button(type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close")
        .modal-body#remote_modal_body
          = yield

As you can see it wraps the modal html within the expected <turbo-frame id="remote_modal"> element with the code…

= turbo_frame_tag "remote_modal" do

The html is almost the standard modal template from bootstrap. But with some modifications…

It expects a title local variable.
It yield on the .modal-body.
It initializes a remote-modal stimulus controller.
It defines an action on the stimulus controller that calls remote-modal#hideBeforeRender.
It defines a #remote_modal_body html id.

At this point the things that are required are the first three…

It expects a title local variable.
It yield on the .modal-body.
It initializes a remote-modal stimulus controller.

Here I am passing the title and putting the yield on the .modal-body just because I think that in this example app it makes scense, but maybe on your app it will be better to just yield on .modal-content and define the .modal-header and .modal-body on the passed block.

The attribute data-controller="remote-modal" initializes the remote modal stimulus controller that on connect will initialize a bootstrap.Modal object for the element and show it.

import { Controller } from "@hotwired/stimulus"
import { Modal } from "bootstrap"

export default class extends Controller {
  connect() {
    this.modal = new Modal(this.element)
    this.modal.show()
  }

  //...
}

And… that’s what you need to show a remote modal…

A link with a data-turbo-frame="remote-modal" (or whatever id you want).
An already defined <turbo-frame id="remote-modal">.
A server response with a matching <turbo-frame id="remote-modal">.
A stimulus controller to “show” the modal, once the content is there.

Handing the form redirect and close the modal

When the form is processed with success we normally want to redirect to a new page. On the example app the redirection is to the products_path, the index page.

Be default a turbo_frame would expect a matching turbo_frame in the response, to update the content of the turbo_frame, but if we want to redirect and close the modal, we need to target the whole page by default with a target="_top".

Like in the template views/layouts/application.html.erb…

<%= turbo_frame_tag "remote_modal", target: "_top" %>

That, combined with the redirect on the create action, will change the content of the whole page and close the modal instantly (without the fade effect).

def create
  @product = Product.new(product_params)

  if @product.save
    redirect_to products_path
  else
    render :form_update, status: :unprocessable_entity
  end
end

Close the modal with the fade effect

If we want to keep the modal’s fade effect that bootstrap provides, we need somehow tell turbo to pause before it renders the page, to let bootstrap hide the modal, and then continue with the render after the modal it is already hidden.

So, you can listen to the turbo:before-render event to then…

Prevent the default behavior.
Close the modal.
Resume the render after the modal has been closed.

In the example app the listener to the turbo:before-render is on the data-action defined on the remote_modal partial…

= turbo_frame_tag "remote_modal" do
  .modal.fade(tabindex="-1" data-controller="remote-modal" data-action="turbo:before-render@document->remote-modal#hideBeforeRender")
    = #...

Here the code is telling stimulus, to listen for the turbo:before-render event on document and then call the remote-modal#hideBeforeRender method.

That method is defined like this…

export default class extends Controller {
  //...

  hideBeforeRender(event) {
    event.preventDefault()
    this.element.addEventListener('hidden.bs.modal', event.detail.resume)
    this.modal.hide()
  }
}

Here the code is…

Preventing the default behavior with event.preventDefault().
Then it sets a listener for the hidden.bs.modal event, that will be triggered by bootstrap after the modal has been closed, and then call event.detail.resume that is provided by turbo.
Then it calls this.modal.hide() that will tell bootstrap to actually hide the modal.

Rendering form errors

To render the errors what you need to do is to respond with a turbo stream that will replace the html of the form.

As you can see on the create action, when @product.save is not successfull, the app will render the form_update template and return the :unprocessable_entity status.

def create
  @product = Product.new(product_params)

  if @product.save
    redirect_to products_path
  else
    render :form_update, status: :unprocessable_entity
  end
end

The form_update template looks like this…

<%= turbo_stream.update "remote_modal_body" do %>
  <%= render "form", product: @product %>
<% end %>

This will respond with a turbo-stream that will look something like this…

<turbo-stream action="update" target="remote_modal_body">
  <template>
    <form action="...">
    </form>
  </template>
</turbo-stream>

This will tell turbo to look for an html id remote_modal_body (defined on the remote_modal partial) and update its content with whatever is inside the template tag. In this case the form with the rendered errors.

Here we don’t actually need to respond with :unprocessable_entity for the errors by rendered. But I think that in this case it is a better response status than 200.

Conclusion

If we try to compact the whole post in some steps…

To show a remote modal you will need…

A link with a data-turbo-frame="remote-modal" (or whatever id you want).
An already defined <turbo-frame id="remote-modal">.
A server response with a matching <turbo-frame id="remote-modal">.
A stimulus controller to “show” the modal, once the content is there.

To handle redirects and close the modal you will need to…

Add target="_top" to your already defined <turbo-frame id="remote-modal">.
Respond with a redirect from the server.

To close the modal with the fade effect…

Listen to the turbo:before-render event and then…
Prevent the default behavior.
Close the modal.
Resume the render after the modal has been closed.

And to render the errors…

Respond with a turbo stream that updates the form html.

Using a form object to create more than one record

2020-10-14T22:54:00Z

Have you been stuck trying to build a form to save more than one record?

Rails form helpers are really easy to work with when you are working with an ActiveRecord::Base object…

But what do you do when you need to create a form to save multiple records?… How do you specify the form_with code and fields?… How do you populate the form options?

One way of doing it is create a new object, what is sometimes called a “form object”.

Here I will try to explain you how you can use a form object to create more than one record using one form.

“Personal finances” demo app

I have build a kind of “personal finances” app , to keep track of transactions in bank accounts.

It has two records, Account and Transaction, and one of the features of the app is that it can register a transfers from one account to another.

The app provides a select for the target account and an input field for the amount, with that it will register two transactions, one “outflow” from the “source account” and one “inflow”, to the “target account”.

Like in the next video…

In this post I will show you how using a “form object”, can help you to…

Specify the fields of the form
Display errors
Populate the “target account” options
Process the received params

Play with the code

The code of the app is on github.com/bhserna/form object_example, you can clone it and play with it.

Specify the fields of the form

If you go to the file of the transfer object you will see that it include ActiveModel::Model and defines some “attribute accessors”…

class Transfer
  include ActiveModel::Model
  attr_accessor :date, :source_account_id, :target_account_id, :amount
  validates_presence_of :date, :source_account_id, :target_account_id, :amount
  #...
end

You don’t really need ActiveModel::Model to build a “form object” but it will help you to handle the attributes initialization and validations like in ActiveRecord::Base objects.

It initializes this object in the transfers controller…

class TransfersController < ApplicationController
  before_action :set_account

  def new
    @transfer = Transfer.new(source_account_id: @account.id)
  end

  #...
end

And it uses it in the form…

= form_with model: @transfer, url: account_transfers_path(@account) do |f|
  = f.hidden_field :source_account_id
  %p
    = f.label :date
    %br
    .date-inputs
      = f.date_select :date, selected: Date.current
  %p
    = f.label :target_account_id
    %br
    = f.collection_select :target_account_id, @transfer.target_options, :id, :name
  %p
    = f.label :amount
    %br
    = f.number_field :amount
    = errors_for @transfer, :amount
  %p
    = f.button

Here the form uses the instance of our Transfer object to populate each field as with regular ActiveRecord::Base objects.

Display errors

If you see the form , you will see that, it uses an errors_for helper…

%p
  = f.label :amount
  %br
  = f.number_field :amount
  = errors_for @transfer, :amount

It is defined in the application helper…

module ApplicationHelper
  def errors_for(model, key)
    tag.div(class: "errors") do
      model.errors.messages_for(key).join(", ")
    end
  end
end

It uses the tag method from ActionView and the errors object provided by ActiveModel::Model in the Transfer object.

Populate the “target account” options

If you see the form again, you will see that for the collection_select it is using @transfer.target_options.

= f.collection_select :target_account_id, @transfer.target_options, :id, :name

You can actually define this method wherever you want. Maybe the Account model could also be a good place. But as I am using this object specifically to handle transfers I think that maybe this could be a good place to put this behavior.

class Transfer
  #...

  def target_options
    Account.where.not(id: source_account_id)
  end
end

Process the received params

On the TransfersController…

It initializes the Transfer with the transfer_params and then it calls @transfer.save.

  def create
    @transfer = Transfer.new(transfer_params)

    if @transfer.save
      #...
    else
      #...
    end
  end

As you can see, It looks almost like an standard scaffold controller!

Constructing the transfer_params

An ActiveRecord::Base object will let us pass directly the params that our date_select returns, but an ActiveModel::Model can’t do that (at least for now).

For that reason to build the transfer_params it first builds the date from the params and then it merges it to the permitted params.

def transfer_params
  params.require(:transfer)
    .permit(:source_account_id, :target_account_id, :amount)
    .merge(date: date_from_params(:transfer))
end

You can find the implementation for the date_from_params method on the ApplicationController.

It looks like this…

def date_from_params(key)
  Date.new(*date_args_from_params(key))
end

def date_args_from_params(key)
  %w(1 2 3).map { |e| params[key]["date(#{e}i)"].to_i }
end

Normalizing values

On the initialization first it transforms the amount from an string to an integer (I am not using cents, because it is a demo, in real life you may want to use a money object or BigDecimal).

def amount=(value)
  @amount = value.to_i if value.present?
end

Saving the form

The Transfer#save method then checks if it is valid and then makes the transfer.

def save
  if valid?
    Transaction.transfer(
      date: date,
      source_account: Account.find(source_account_id),
      target_account: Account.find(target_account_id),
      amount: amount
    )
  end
end

Here, it using the valid? method that will executes the defined validations, provided by ActiveModel::Model.

The Transaction.transfer method is implemented on Transaction, just because I think that it makes more scence there. But depending on the complexity of your problem you can execute the creation of the records directly on the form or delegate it to another object,

class Transaction < ApplicationRecord
  #...

  def self.transfer(date:, source_account:, target_account:, amount:)
    transaction do
      create(
        date: date,
        account: source_account,
        target_name: target_account.name,
        description: "Transfer to #{target_account.name}",
        amount: -1 * amount
      )

      create(
        date: date,
        account: target_account,
        target_name: source_account.name,
        description: "Transfer from #{source_account.name}",
        amount: amount
      )
    end
  end
end

Conclusion

Cool I think that’s it!!

As you can see, you can use a form object when you need to create more than one record by using just one form to capture the data. It can help you to…

Specify the fields of the form
Display errors
Compute custom data for the form, like select options
Process the received params

Remember that for now if your form uses date_select or datetime_select you will have to build the date from the params before passing it to your form object.

Dynamic filters with rails and hotwire

2021-12-14T00:40:00Z

A very common task as a Rails developer is to let the user filter a list with a combination of search fields and selects.

Here I want to show you one way of doing it using turbo and stimulus.js from hotwire, with an example app.

Play with the code

The code of the app is on github.com/bhserna/dynamic filters hotwire, you can clone it and play with it.

What does the app do?

The app displays a paginated list of products and lets the user filter the products by selecting a category and typing a search term.

How does it work?

By watching the file template products/index.haml, you will see that the index page is composed of the title and two partials, filters and table.

%h1 Products

= render "filters"
= render "table", products: @products, pagy: @pagy

It is not necessary to break the html in this two partials but I like it that way 😅. It helps me to understand the code.

The “special” thing of the app is that it uses turbo and stimulus.js to update the content when of the table when you select a new category or if you type in the search field.

The filters form

If you go to the products/filters partial you will see that the form defines three important data attributes…

data-controller="filters"
data-filters-target="form"
data-turbo-frame="table"

= form_with(url: products_path,
  class: "filters-container",
  method: :get,
  data: { controller: "filters", filters_target: "form", turbo_frame: "table" }) do |f|

data-controller="filters" tells stimulus.js to create a instance of the controller class in filters_controller.js.

data-filters-target="form" marks the form as a formTarget in the filters_controller.js.

data-turbo-frame="table tells turbo that you want to update turbo-frame with the id table. If you don’t set this, turbo will change the whole page and if the user is typing in the search input it will lose focus.

The inputs

In the same form the select and the input…

.filters-group
  = f.label :category, "Filter by category"
  = f.select :category,
    options_for_select(Product.categories.keys, params[:category]),
    { include_blank: true },
    { data: { action: "filters#submit" } }

.filters-group
  = f.label :search, "Search"
  = f.text_field :search,
    { value: params[:search], data: { action: "filters#submit" } }

… both have a data-action="filters#submit" that will call the submit() function on the filters_controller.js.

import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  static get targets() {
    return ["form"]
  }

  submit(event) {
    this.formTarget.requestSubmit()
  }
}

To submit the form we need to actually trigger the submit event that’s why need to call requestSubmit() instead of submit().

The table partial

Now if you go to the products/table you will find that the first line wraps the rest of the content in a turbo_frame_tag with an id "table".

= turbo_frame_tag "table" do
  %div.loading-container
    %div.loading-element Loading...
  %div.table-container
    %table
      %thead
        %tr
          %th Name
          %th Category
          %th Price
      %tbody
        = render products
  = raw pagy_nav(@pagy)

So, if you watch the html output you will see…

<turbo-frame id="table">
  # rest of the content...
</turbo-frame>

This tag matches the data-turbo-frame="table" from the form, and now turbo knows that it should update this “turbo frame” with the server response when the form is submitted.

The loading indicator

If you wathch the video again, you will see that there is a little “Loading…” that is shown when the form changes.

The html that renders this indicator is on the products/table partial, specifically this part…

= turbo_frame_tag "table" do
  %div.loading-container
    %div.loading-element Loading...

The css for the class .loading-element is defined on the application.css as…

turbo-frame .loading-element { display: none; }
turbo-frame[busy] .loading-element { display: block; }

This works because busy is a boolean attribute managed by turbo. It is toggled to be present when a turbo-frame-initiated request starts, and toggled false when the request ends

Conclusion

As you can see with very little “special code” hotwire can help us to achieve this very common task.

What you need to make this work is to set data-turbo-frame="table" on a form and have a turbo_frame_tag "table" on the same page, and call requestSubmit() when you want to submit the form.

If you want a loading indicator, you can use the turbo_frame[busy] attribute to display the indicator when needed.

Styler, a tool to compose css classes with ruby

2021-12-07T12:36:00Z

One of the things that I want from css is to have to possibility to compose already defined styles, to define new ones…

If you try to write “Semantic CSS”, you will find a hard time trying to avoid the repetition on things that look the same but are different things, like when you want to render a “card” for the an author and then for an article.

You can create “content agnostic CSS components”, but things start to get complicated, when you want to avoid duplication if things from one component are similar to other components.

One way of solving this problems is by using something like the @extend function from sass, or the @apply function from Tailwind css, but both tools are recommended just for some specific cases and not to build your styles on top of them (@extend, @apply).

A tool to compose css classes

This is why I have been working on this Styler, a tool to compose css classes from other classes an groups of classes.

Is a tool, to extend what the utility css classes can give us. A tool to build on top of frameworks like Tachyons or Tailwind css.

Define styles

You can declare styles with a name and an already defined css class, for example if you define a style named btn, like this…

styles = Styler.new do
  style :btn, ["white", "bg_blue"]
end

That will return an object where you can call btn on it…

styles.btn.to_s # => "white bg_blue"

You would be able to use it like this in your erb files…

<button class="<%= styles.btn %>">My button</button>

or in haml…

%button{class: styles.btn} My button

To output…

<button class="white bg_blue">My button</button>

Compose styles

You can define many of this styles and compose them…

styles = Styler.new do
  style :btn, ["padding3", "marin3"]
  style :blue_btn, [btn, "white", "bg_blue"]
end

%button{class: styles.btn} My button
%button{class: styles.blue_btn} My button

And the output would be…

<button class="padding3 margin3">My button</button>
<button class="padding3 margin3 white bg_blue">My button</button>

Substract styles

By composing your styles you can also substract classes from previous styles, like this…

styles = Styler.new do
  style :default, ["pa3", "white", "bg_blue"]
  style :danger, [default - "bg_blue", "bg_red"]
end

styles.default.to_s # => "pa3 white bg_blue"
styles.danger.to_s # => "pa3 white bg_red"

Passing arguments to styles

You can also define styles that expect an argument, to help you decide which styles to display, like this…

styles = Styler.new do
  style :default_color do |project|
    if project[:color] == "blue"
      ["bg_blue"]
    else
      ["bg_red"]
    end
  end
end

project = { color: "blue" }
styles.default_color(project).to_s # => "bg_blue"

And you can use this styles to build other styles, like this…

project = { color: "blue" }

styles = Styler.new do
  style :default_color do |project|
    if project[:color] == "blue"
      ["bg_blue"]
    else
      ["bg_red"]
    end
  end

  style :title, [default_color(project), "pa3"]
end

styles.title(project).to_s # => "bg_blue pa3"

Or like this…

styles = Styler.new do
  style :default_color do |project|
    if project[:color] == "blue"
      ["bg_blue"]
    else
      ["bg_red"]
    end
  end

  style :title do |project|
    [default_color(project), "pa3"]
  end
end

project = { color: "blue" }
styles.title(project).to_s # => "bg_blue pa3"

Define collections

You can define collections as namespaces for your styles…

styles = Styler.new do
  collection :buttons do
    style :default, ["pa3", "blue"]
    style :danger, [default - "blue", "red"]
  end
end

styles.respond_to?(:default) # => false
styles.respond_to?(:danger) # => false
styles.buttons.default.to_s # => "pa3 blue"
styles.buttons.danger.to_s # => "pa3 red"

Nested collections

You can define nested collections to build complete themes…

styles = Styler.new do
  collection :v1 do
    collection :buttons do
      style :default, ["pa3", "blue"]
    end
  end

  collection :v2 do
    collection :buttons do
      style :default, ["pa3", "red"]
    end
  end
end

styles.v1.buttons.default.to_s # => "pa3 blue"
styles.v2.buttons.default.to_s # => "pa3 red"

Define collection with arguments

Like with the styles, you can define collections that require arguments, like this..

styles = Styler.new do
  collection :buttons do |project|
    if project[:color] == "blue"
      style :default, ["pa3", "blue"]
    else
      style :default, ["pa3", "red"]
    end
  end
end

project = { color: "blue" }
styles.buttons(project).default.to_s # => "pa3 blue"

Select a collection with an alias

And you can pick one of those collections, by using a collection_alias…

styles = Styler.new do
  collection :v1 do
    collection :buttons do
      style :default, ["pa3", "blue"]
    end
  end

  collection :v2 do
    collection :buttons do
      style :default, ["pa3", "red"]
    end
  end

  collection_alias :theme, v1
end

styles.theme.buttons.default.to_s # => "pa3 blue"

Collection alias with a block

If you need you can select the collection alias dynamically with a block…

styles = Styler.new do
  collection :v1 do
    collection :buttons do
      style :default, ["pa3", "blue"]
    end
  end

  collection :v2 do
    collection :buttons do
      style :default, ["pa3", "red"]
    end
  end

  collection_alias :theme do |current_version|
    if current_version == "v1"
      v1
    else
      v2
    end
  end
end

styles.theme("v1").buttons.default.to_s # => "pa3 blue"
styles.theme("v2").buttons.default.to_s # => "pa3 red"

Select a collection from other styler

You can also define your collections on different “stylers” and be able to declare an alias to access them…

v1 = Styler.new do
  collection :buttons do
    style :default, ["pa3", "blue"]
  end
end

v2 = Styler.new do
  collection :buttons do
    style :default, ["pa3", "red"]
  end
end

styles = Styler.new do
  collection_alias :theme do |current_version|
    if current_version == "v1"
      v1
    else
      v2
    end
  end
end

styles.theme("v1").buttons.default.to_s # => "pa3 blue"
styles.theme("v2").buttons.default.to_s # => "pa3 red"

Copy styles from collection

If you need to use the styles from other collection, but you need to override some of them, you can copy the styles and then override what you want.

styles = Styler.new do
  collection :v1 do
    collection :buttons do
      style :default, ["pa3", "blue"]
      style :danger, [default - "blue", "red"]
    end
  end

  collection :v2 do
    collection :buttons do
      copy_styles_from collection: v1.buttons
      style :danger, [v1.buttons.danger - "red", "orange"]
    end
  end
end

expect(styles.v2.buttons.default.to_s).to eq "pa3 blue"
expect(styles.v2.buttons.danger.to_s).to eq "pa3 orange"

Read the source, Luke

If you want to learn more you can read the source on github, and maybe take a look a the specs to find more examples.

Library status… It would be nice to receive some feedback.

I have been working on this tool for some weeks, and I have been using it on some personal projects and to build a collection with the tachyons components.

I have pitched the library to my team at briq.mx (we use Tachyons), but we are not using it yet…

Is still a concept, and would be nice to receive some feedback from you!

Installation

If you want to play with this tool in a project you can install it by adding this line to your application’s Gemfile:

gem "ruby_styler"

And then execute:

$ bundle install

Or install it yourself as:

$ gem install styler

Capybara cheatsheet as pdf

2021-11-29T13:27:00Z

If you work with capybara and you are constantly searching for capybara helpers, maybe to have this little cheatsheet at hand could work for you.

I already have share it on as a blog post, but now I want to share it with you as a pdf.

Preloading associations cheatsheet

2021-11-19T12:32:00Z

Maybe you are already familiar with includes or preload, but you know that a lot of the time you will need more than just preload(:comments).

I have already share with you a guide for preloading associations…

It starts with the basics, with just a regular has_many, preload orincludes, and build from this to then show you things like…

How to preload nested associations
How to define associations to help you simplify the preloading
How to preload just a part of the association like an scope
How to preload just the “top 1 per group” or the “latest of each”
How to preload just the “top n per group” or the “latest n of each”
How to use custom objects to represent a preloading

Now I want to share with you a little cheatsheet that covers the same but in a condensed way.

It could be useful as a first read of the content or as a tool to remember hints of the different topics or methods.

How to detect n+1 queries by watching the logs

2021-11-16T12:56:00Z

Although there are tools that can help you detect n+1 queries before they hit production, I think that is good to be able to identify n+1 queries directly by watching your logs…

Not all the tools will tell you exactly that you have n+1 queries, and sometimes the ones that will tell you, can give you false negatives.

Also I think that, although there is hardly no training anywhere for this ability, is always an expected ability for an experienced rails developer.

Look for repeated queries

As you may know, an n+1 queries problem means that a query is executed for every result of a previous query.

For example, given a Post that has many comments, If you execute the next code…

Post.map(&:comments).map(&:to_a)

You will execute…

1 query to fetch the posts (Post.all)
And n queries to fetch the comments of each post(post.comments)

So, if you go to the console and test this code…

If there are 2 posts, it will execute 3 queries (2 + 1), and you will see in your logs, something like…

> Post.first(2).map(&:comments).map(&:to_a)
Post Load (0.8ms)  SELECT "posts".* FROM "posts" ORDER BY "posts"."id" ASC LIMIT $1  [["LIMIT", 2]]
Comment Load (0.4ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 1]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 2]]

And if there are 5 posts, it will execute 6 queries (5 + 1), and you will see in your logs, something like…

> Post.first(5).map(&:comments).map(&:to_a)
Post Load (0.8ms)  SELECT "posts".* FROM "posts" ORDER BY "posts"."id" ASC LIMIT $1  [["LIMIT", 5]]
Comment Load (0.4ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 1]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 2]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 3]]
Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 4]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 5]]

And if there are 1000 posts, it will execute 1001 queries (1000 + 1), and in the same way you will see in your logs…

> Post.first(1000).map(&:comments).map(&:to_a)
Post Load (0.8ms)  SELECT "posts".* FROM "posts" ORDER BY "posts"."id" ASC LIMIT $1  [["LIMIT", 1000]]
Comment Load (0.4ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 1]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 2]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 3]]
Comment Load (0.2ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 4]]
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 5]]
#...
Comment Load (0.3ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" ASC  [["post_id", 1000]]

Look for repeated queries executed in the same place

The previous examples show examples taken from the rails console, but as you can see in the next example…

When you start the server, on your logs, after each query, rails will show you where is the code that executes that query.

Started GET "/posts" for ::1 at 2021-11-10 06:47:40 -0600
Processing by PostsController#index as HTML
  Rendering layout layouts/application.html.erb
  Rendering posts/index.html.erb within layouts/application
  Rendered posts/_filters.html.erb (Duration: 6.5ms | Allocations: 2354)
  Post Load (1.2ms)  SELECT "posts".* FROM "posts" WHERE "posts"."comments_count" = $1  [["comments_count", 5]]
  ↳ app/views/posts/index.html.erb:5
  Comment Load (3.4ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" DESC LIMIT $2  [["post_id", 1], ["LIMIT", 1]]
  ↳ app/views/posts/_post.html.erb:6
  Comment Load (1.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" DESC LIMIT $2  [["post_id", 2], ["LIMIT", 1]]
  ↳ app/views/posts/_post.html.erb:6
  Comment Load (1.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" DESC LIMIT $2  [["post_id", 3], ["LIMIT", 1]]
  ↳ app/views/posts/_post.html.erb:6
  Comment Load (1.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" DESC LIMIT $2  [["post_id", 4], ["LIMIT", 1]]
  ↳ app/views/posts/_post.html.erb:6
  Comment Load (1.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = $1 ORDER BY "comments"."id" DESC LIMIT $2  [["post_id", 5], ["LIMIT", 1]]
  ↳ app/views/posts/_post.html.erb:6
  #...
  Rendered collection of posts/_post.html.erb [20 times] (Duration: 84.5ms | Allocations: 34947)
  Rendered posts/index.html.erb within layouts/application (Duration: 107.3ms | Allocations: 45186)
  Rendered layout layouts/application.html.erb (Duration: 175.0ms | Allocations: 77162)
Completed 200 OK in 204ms (Views: 147.6ms | ActiveRecord: 34.7ms | Allocations: 86545)

So, if you look out for this two things, repeated queries that are executed from the sample place … you can be almost sure that you have an n+1 queries problem in that place.

How can I determine if my n+1 queries fix can hurt the performance?

2021-11-02T12:56:00Z

Maybe you already know that “fixing” an n+1 queries problem can hurt the performance of your app…

But even if you already know it and also understand how it can happen, it could not be obvious how to know if this is will happen with your fix.

Some things that you can try are to…

Compare the requests time
Compare the queries load time
Compare the produced allocations

Depending on the criticality of the fix and the volume that is handling your app you can just push to production and check there or try to simulate with the expected volume of data on development.

Try on development first

Although sometimes there will be no problem if you just push to production, if your intuition tells you that maybe the associations that you are preloading will preload to much data, you can try to simulate the expected volume of data that you will expect on production. This will help you to have more control on your tests.

Where to find the information?

You can use different tools like rack-mini-profiler or tools on your production application, like NewRelic or Skylight, but you can also take a look on your logs.

If you looks in your logs, for each request you will find something like this…

Started GET "/posts" for ::1 at 2021-10-05 18:11:18 -0500
Processing by PostsController#index as HTML
  Rendering layout layouts/application.html.erb
  Rendering posts/index.html.erb within layouts/application
  Post Load (4.2ms)  SELECT "posts".* FROM "posts" WHERE "posts"."comments_count" = $1  [["comments_count", 5]]
  ↳ app/views/posts/index.html.erb:35
  Comment Load (4.8ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20) ORDER BY "comments"."id" ASC  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4], ["post_id", 5], ["post_id", 6], ["post_id", 7], ["post_id", 8], ["post_id", 9], ["post_id", 10], ["post_id", 11], ["post_id" 12], ["post_id", 13], ["post_id", 14], ["post_id", 15], ["post_id", 16], ["post_id", 17], ["post_id", 18], ["post_id", 19], ["post_id", 20]]
  ↳ app/views/posts/index.html.erb:35
  Rendered posts/index.html.erb within layouts/application (Duration: 58.0ms | Allocations: 18269)
  Rendered layout layouts/application.html.erb (Duration: 64.0ms | Allocations: 18907)
Completed 200 OK in 92ms (Views: 50.9ms | ActiveRecord: 23.8ms | Allocations: 24737)

There, for the sql you will find…

The sql used for the query
The query load time
And the place where the query was called in your code

Post Load (4.2ms)  SELECT "posts".* FROM "posts" WHERE "posts"."comments_count" = $1  [["comments_count", 5]]
  ↳ app/views/posts/index.html.erb:35
  Comment Load (4.8ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20) ORDER BY "comments"."id" ASC  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4], ["post_id", 5], ["post_id", 6], ["post_id", 7], ["post_id", 8], ["post_id", 9], ["post_id", 10], ["post_id", 11], ["post_id" 12], ["post_id", 13], ["post_id", 14], ["post_id", 15], ["post_id", 16], ["post_id", 17], ["post_id", 18], ["post_id", 19], ["post_id", 20]]
  ↳ app/views/posts/index.html.erb:35

For each rendered file you will find…

The rendering time as “Duration”
And the number of Allocations

Rendered posts/index.html.erb within layouts/application (Duration: 58.0ms | Allocations: 18269)
Rendered layout layouts/application.html.erb (Duration: 64.0ms | Allocations: 18907)

And at the bottom you will find the total times…

The request time
The time spent in views
The time spent in ActiveRecord
And the time in Allocations

Completed 200 OK in 92ms (Views: 50.9ms | ActiveRecord: 23.8ms | Allocations: 24737)

Practice app

I have built a little app to help you visualize why “fixing” an n+1 queries problem, can hurt the performance of your application if you preload associations with too many records.

It simulates the index page of a blog application. It has the rack-mini-profiler gem, and it shows the logs in the page to help you visualize easily the queries, request time and allocations.

By default it has an n+1 queries problem. It makes a call to the database to fetch the latest comment for each post in the list, but you can fix it with a checkbox.

It will let you compare request times combining a filter of posts by number of comments and the checkbox to “fix” the n+1 queries problem.

I explain more about it on this link: Example app to understand why some times fixing some n+1 queries can hurt performance

Joins does not preload

2021-10-24T12:56:00Z

When you are working with SQL, to use information from other tables you would use a JOIN…

And if you are just starting to learn ActiveRecord comming from SQL, you could think that by using the joins method you will be able to later use the data from the associations without doing a new query.

And this is true and false at the same time…

It will let you use the association on the query

Yes you will be able to use the data of the associations but just in the same query. If you want to use the data from an association after the query result has been returned, you will need a new query.

To see it more clearly let’s try an example…

This will do an INNER JOIN:

posts = Post.joins(:comments)

And this will trigger a new database call for each post:

posts.map(&:comments).to_a

Because joins does not preload the data.

If you see the produced query you will see that what rails does, is to use the association data to build the join to the comments table.

SELECT posts.*
FROM posts
INNER JOIN comments ON comments.post_id = posts.id

With that join you will be able to use the associated relation (in this case the comments table) in the query, by chaining other methods from the query interface like a where, order, group…

For example…

posts = Post
  .joins(:comments)
  .where(comments: { body: "hola" })

This will build the SQL clause with a WHERE like this:

SELECT posts.*
FROM posts
INNER JOIN comments ON comments.post_id = posts.id
WHERE comments.body = 'hola'

You can do the same with other methods like order:

posts = Post
  .joins(:comments)
  .order("comments.created_at DESC")

This will build the SQL clause with an ORDER like:

SELECT posts.*
FROM posts
INNER JOIN comments ON comments.post_id = posts.id
ORDER BY comments.created_at DESC

To preload you will need another method

And if you really want to preload the comments you will need to use preload , includes or eager_load, to actually tell ActiveRecord to also fetch the associated records. For example:

This will do an INNER JOIN and a also preload the comments:

posts = Post
  .joins(:comments)
  .where(comments: { body: "hola" })
  .preload(:comments)

And now this will not trigger a new database call for each post:

posts.map(&:comments).to_a

More about joins and methods to preload

If you want learn more about the difference between joins and the three method that you can use to preload associations preload, includes and eager_load. You can read this post:

What is the difference between includes, preload, eager_load and joins in ActiveRecord?

Example app to understand why some times fixing some n+1 queries can hurt performance

2021-10-12T12:56:00Z

I want to share with you an example application to help you visualize why “fixing” an n+1 queries problem, can hurt the performance of your application if you preload assocations with too many records.

It simulates the index page of a blog application. And appart from the rack-mini-profiler gem, it shows the logs in the page to help you visualize easily the queries, request time and allocations.

By default it has an n+1 queries problem. It makes a call to the database to fetch the latest comment for each post in the list, but you can fix it with a checkbox.

It will let you compare request times combining a filter of posts by number of comments and the checkbox to “fix” the n+1 queries problem.

Run the app

You can access the code on github, on bhserna/why_n_plus_one_fix_hurts_performance.

The app uses:

ruby-3.0.0
Postgres
Redis

To use the app effectively you need to run the seeds.rb because it will create the posts that you need to test the app.

seeds = Seeds.new
seeds.create_posts 20, comments_count: 5
seeds.create_posts 20, comments_count: 10
seeds.create_posts 20, comments_count: 50
seeds.create_posts 20, comments_count: 100
seeds.create_posts 20, comments_count: 1000
seeds.create_posts 20, comments_count: 10000

(Don’t worry it is semi-fast because it uses insert_all)

You can update that file to test other combinations.

To create and initialize the database for the first time you can use:

./bin/rails db:setup

If you want to reset the database and run new seeds, you can use:

./bin/rails db:reset

Fix n+1 queries

This is code to display each post…

<% @posts.each do |post| %>
  <div class="post">
    <h1><%= post.title %></h1>
    <p>💬 <%= number_with_delimiter post.comments.size %>

    <div class="comment">
      <p><strong>Latest comment:</strong> <%= post.comments.last.body %></p>
    </div>
  </div>
<% end %>

As you can see… if we don’t preload the comments, ActiveRecord will make a new database call for each post in this line…

<p><strong>Latest comment:</strong> <%= post.comments.last.body %></p>

The checkbox with label “Fix n+1 queries?” will fix the n+1 queries doing preload(:comments) on the controller.

def index
  @posts = Post.where(comments_count: current_filter)

  if fix_n_plus_one_queries?
    @posts = @posts.preload(:comments)
  end
end

Filter the posts by number of comments

In the page we have a select field that will display just the posts with the selected number of comments. We have posts with 5, 10, 50, 100, 1,000 and 10,000.

For each “number of comments” option we have 20 posts.

You can use this select to play with different combinations of number of comments, with and without the fix.

Finding the request time and allocations

You can see the request time and allocations on the logs, if you look for something like this:

Completed 200 OK in 92ms (Views: 50.9ms | ActiveRecord: 23.8ms | Allocations: 24737)

You can also see query times:

Post Load (4.2ms) SELECT "posts".* FROM "posts"....

And template rendering times:

Rendered posts/index.html.erb within layouts/application (Duration: 58.0ms | Allocations: 18269)

Compare the request time

If you compare different combiantions of number of comments, you will see that “fixing” the n+1 queries problem, sometimes will produce faster responses, but not always.

And for some cases, even when it is faster it could not be enough for your needs, and maybe is time to look for other ways to solve the problem.

Compare the allocations

Nate Berkopec has a rule of thumb…

<10,000 = don’t sweat it

10k->100,000k = might matter if you loop, but also don’t worry about it

100k->1mil = reducing objects would speed this up considerably

1mil->10mil = This is increasing your process’ memory + is very slow

>10mil = ☠️

You can play with the app and check the number of allocations that it is producing for each request with differente combinations.

How to fix your n+1 queries fix

If are in the situation where your fix is not really a fix or that is not good enough. There are several ways in that you can fix your problem.

For a lot of this cases the first step is to implement a counter_cache but when you really need to preload the data because you need to something more than a counter, you can give a look to:

Why some people say that fixing some n+1 queries could hurt performance?

2021-10-08T12:56:00Z

Sometimes people say that “fixing” some n+1 queries could hurt performance…

Maybe this phrase can be confusing, because if you have not been exposed to a lot of n+1 queries problems, it could be hard to imagine how it can be possible.

And also is probably contrary to what you have always heard…. That n+1 queries are bad.

You can preload too much data

Although a lot of the time a simple preload(:comments) can solve your n+1 queries problems… sometimes it can also hurt the performance if you don’t put attention in the queries that ActiveRecord is using.

Sometimes some of those queries will try to fetch a lot of data and that can make your queries slow or they can use to much memory and affect your application in general.

For example if you have a blog app with this model…

class Post < ApplicationRecord
  has_many :comments
end

class Comment < ApplicationRecord
end

And you preload all the comments for each post…

posts = Post.preload(:comments)

If your posts, would normally have just a few comments, like 5 to 100 comments, everything will be fine….

But if your posts are very popular, and have between 5,000 and 10,000 maybe “just” preloading all the comments for each post could be a problem.

Be aware of associations with too many records

Maybe this example is not the most realistic example, but in general you need to be aware if the association that you want to preload could have a lot of records.

Other examples could be…

In a chat app:

Preloading all the messages for all the channels.
Preloading all the reads for the messages on all the channels.
Preloading all the reactions to the messages.

In an store or inventory system:

Preload all the part_numbers for a list of products.
Preload all the page_views for a list of products.
Preload all the likes for a list of products.

In a crowdfunding system:

Preload all the investments for all the projects.
Preload all the payments for all the investments for all projects.

In general not all associations will be a problem, but be aware if your application have associations with too many records.

How to fix your n+1 queries fix

If your are in the situation where your fix was not really a fix or it is not good enough. Here are some things that you can try…

First, of all for a lot of this cases the first step is to implement a counter_cache.

And when you really need to preload the data because you need something more than a counter, you can give a look to:

Preload object on the guide for preloading associations

2021-09-13T12:56:00Z

I want to tell you that I have added a new section to the guide for preloading associations in rails, to introduce you to something I call “Preload objects” that will help you build complex preloads for those cases when you can’t find a way to do what you need to do, with the standard rails mechanisms.

Visit the new section

You can visit the new section on Guide for preloading associations in rails - Preload Object. And a standalone article about it on Preload objects.

If you don’t know what this guide is about…

Is a guide I wrote to try to help you get better on this specific skill of preloading associations, for when you need a more than just a preload(:comments).

To help you tackle problems like complex nested associations, scopes that you need or want to reuse, places where preloading all the associated records could hurt the performance of your app.

I hope it can help you =)

Using an objet to represent a complex preload

2021-08-23T12:56:00Z

Sometimes just using preload or includes is not possible… or maybe it is, but you just can’t figure it out how to do it.

Maybe you want to preload some records matching two keys, or preload a grouped relation.

Maybe you know how to represent the association with a has_many :through, but you need something faster, or to use less memory.

In that kind of situations, one thing that you can do is to write a custom object to represent that preload.

Here I will try to explain how and when you can do it with an example.

Example

For example, given the next model:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
  has_many :comment_votes, foreign_key: :voter_id
end

Imagine that you want to fetch the people that have a vote on a comment to a post, the “comment voters”.

To implement it, you can have a method in the post to do that…

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments

  def comment_voters
    comments.preload(votes: :voter)
      .flat_map(&:votes)
      .flat_map(&:voter)
      .uniq
  end
end

And it will correctly preload the votes and voters to compute the result.

post = Post.first
post.comment_voters

But if you try to do it for a list of post, it will execute n+1 queries… What could you do?

Preload with has_many :through

In this particular case you can try to re-shape your associations to solve it with a has_many :through.

You can add a has_many :voters to Comment and a has_many :comment_voters to Post.

Like this:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
  has_many :comment_voters, -> { distinct }, through: :comments, source: :voters
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
  has_many :voters, through: :votes
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
end

This will correctly compute the data in just one query

post = Post.first
post.comment_voters

And, you will also be able to preload the comment_voters just as any other association.

Post.preload(:comment_voters).all.each do |post|
  puts post.comment_voters.to_a
end

But…

If you see the executed queries, it could be not as performant as you need…

irb(main):055:0> Post.preload(:comment_voters).to_a.count

Post Load (0.7ms)  SELECT "posts".* FROM "posts"

Comment Load (3.5ms)  SELECT "comments".* FROM "comments" WHERE
"comments"."post_id" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4],
["post_id", 5], ["post_id", 6], ["post_id", 7], ["post_id", 8], ["post_id", 9],
["post_id", 10], ["post_id", 11], ["post_id", 12], ["post_id", 13], ["post_id",
14], ["post_id", 15], ["post_id", 16]...

CommentVote Load (18.3ms)  SELECT "comment_votes".* FROM "comment_votes" WHERE
"comment_votes"."comment_id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74,
75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94,
95, 96, 97, 98, 99, 100...

User Load (0.2ms)  SELECT DISTINCT "users".* FROM "users" WHERE "users"."id" IN
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)  [["id", 8], ["id", 5], ["id", 1], ["id", 6],
["id", 4], ["id", 7], ["id", 3], ["id", 10], ["id", 2], ["id", 9]]

Maybe you can try to eager_load, but maybe it could not be what you need either…

irb(main):059:0> posts = Post.eager_load(:comment_voters).to_a

SQL (47.0ms)  SELECT "posts"."id" AS t0_r0, "posts"."author_id" AS t0_r1,
"posts"."title" AS t0_r2, "posts"."body" AS t0_r3, "posts"."created_at" AS
t0_r4, "posts"."updated_at" AS t0_r5, "users"."id" AS t1_r0, "users"."name" AS
t1_r1, "users"."created_at" AS t1_r2, "users"."updated_at" AS t1_r3 FROM
"posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" LEFT
OUTER JOIN "comment_votes" ON "comment_votes"."comment_id" = "comments"."id"
LEFT OUTER JOIN "users" ON "users"."id" = "comment_votes"."voter_id"

Trying a custom query

Maybe what you want is to write a custom query, because you don’t want to fetch data that you don’t need.

Let’s imagine that what you need is to fetch just the users with voted comments on a collection of posts. Without preloading anything else, like this:

posts = Post.limit(30)
comment_voters = User
  .select("users.*, comments.post_id") # selects the user attributes and the post_id of the comment.
  .joins(comment_votes: [:comment])
  .distinct
  .where(comments: { post_id: posts })
  .to_a

User Load (16.5ms)  SELECT DISTINCT users.*, comments.post_id FROM "users"
INNER JOIN "comment_votes" ON "comment_votes"."voter_id" = "users"."id" INNER
JOIN "comments" ON "comments"."id" = "comment_votes"."comment_id" WHERE
"comments"."post_id" IN (SELECT "posts"."id" FROM "posts")

You can then group this voters by post_id, and then in the view you can pick the voters for each post.

posts = Post.limit(30)
comment_voters = User
  .select("users.*, comments.post_id") # selects the user attributes and the post_id of the comment.
  .joins(comment_votes: [:comment])
  .where(comments: { post_id: posts })
  .distinct
  .group_by(&:post_id)

posts.each do |post|
  comment_voters[post.id].map(&:name)
end

That’s ok, but…

What if you don’t want to expose that code in your controller or view?… What if instead we move that to a model object?… Where should you put that code?

A preload object

There are many options, but one of them, is to create a new object to help you with two things:

Fetch the voters for the posts in one call to the database
Return the voters for a given post

For this example you can call the object Post::CommentVotersPreload and use it like this:

posts = Post.limit(30)
comment_voters = Post::CommentVotersPreload.new(posts)

posts.each do |post|
  comment_voters.for_post(post).map(&:name)
end

And the implementation could look something like this:

class Post::CommentVotersPreload
  def initialize(posts)
    @posts = posts
  end

  def for_post(post)
    comment_voters[post.id]
  end

  private

  def comment_voters
    @comment_voters ||= fetch_records.group_by(&:post_id)
  end

  def fetch_records
    User.select("users.*, comments.post_id")
      .joins(comment_votes: [:comment])
      .where(comments: { post_id: @posts })
      .distinct
  end
end

You can use this kind of objects in many different problems.

What you will need is:

A fetch_records method to fetch the records you want to preload in one call to the database (or at least not n+1 queries).
A find_for method to return the records related to each associated object.

Other examples

If you need more examples to understand the idea, you can check this other examples:

When should you use it?

The answer to this question is not that simple (at least for me).

What I could recommend you is to first try to solve the problem in a “railsy” way… Try to use the mechanisms, provided by rails. Normally those mechanisms will compose better with other parts of the framework.

If you don’t know how to do it, because you don’t know enough about rails, try to study the rails guides, specially the associations guide, and the query interface guide. You can also study my guide for preloading associations.

But if you already know what rails provides, and you can’t find a way to do what you need to do… or you need something more efficient… or you just think that the final code is better…

Then go for it and use a “preload object”, just try to have a reason.

A guide for preloading associations in rails

2021-08-09T12:56:00Z

Maybe you are already familiar with includes or preload, but you know that a lot of the time you will need more than just preload(:comments).

It is common to have complex nested associations, scopes that you need or want to reuse, places where preloading all the associated records could hurt the performance of your app.

How do you work with complex nested associations?… How do you simplify the preloading of those nested associations?… How do you “preload an scope”?… How do you preload just the latest n of each record?

ActiveRecord gives us some ways to do some of those things, but some of them are not in the documentation or is not that easy to find them.

Well, this is a guide to help you get better on this specific skill of preloading associations, for when you need a more than just a preload(:comments).

It starts with the basics, with just a regular has_many, preload or includes, and build from this to then show you things like…

How to preload nested associations
How to define associations to help you simplify the preloading
How to preload just a part of the association like an scope
How to preload just the “top 1 per group” or the “latest of each”
How to preload just the “top n per group” or the “latest n of each”
How to use custom objects to represent a preloading

Index

Preload direct associations
Preload nested associations
Preload an scope
Preload the top 1 of an association
Preload the top n of an association
Simplify preloading with has_many through
Preload object

Preload direct associations

You can preload direct associations preload, includes or eager_load. They are very similar but with some differences.

preload - Makes a new query to preload the associations
eager_load - It forces the eager loading performing a LEFT OUTER JOIN.
includes - By default works like preload, but in some cases it will behave like eager_load, normally when you are also adding some conditions to the query.

If you want to know the difference in more detail you can see What is the difference between includes, preload, eager_load and joins in ActiveRecord?

For example given the next model:

class Post < ApplicationRecord
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
end

You will be able to fetch the posts, preloading the comments doing:

posts = Post.preload(:comments)
# or...
posts = Post.includes(:comments)
# or...
posts = Post.eager_load(:comments)

And if you ask for the comments of each post it will not require a new call to the database.

puts posts.map { |post| post.comments.to_a }

You will be able to fetch the comments, preloading the posts doing:

comments = Comment.preload(:post)
# or...
comments = Comment.includes(:post)
# or...
comments = Comment.eager_load(:post)

And in the same way, if you ask for the post of each comment it will not require a new call to the database.

puts comments.map { |comment| comment.post }

Preload nested associations

For nested associations you can pass a hash with the root association as the key and the associations that you want to preload as an array.

The array can include more nested associations.

For example, given the next model:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
end

You can fetch the posts preloading comments and users:

posts = Post.preload(comments: :user)

puts posts.flat_map(&:comments).map(&:user)

Fetch comments preloading posts with author:

comments = Comment.preload(post: :author)

puts comments.map(&:post).map(&:author)

Fetch posts preloading comments with users and votes

posts = Post.preload(comments: [:user, :votes])

puts posts.flat_map(&:comments).map(&:user)
puts posts.flat_map(&:comments).map(&:votes).map(&:to_a)

Fetch posts preloading comments with users, votes and voters

posts = Post.preload(comments: [:user, votes: [:voter]])

puts posts.flat_map(&:comments).map(&:user)
puts posts.flat_map(&:comments).flat_map(&:votes).map(&:voter)

Fetch posts preloading author, comments and users

posts = Post.preload(:author, comments: :user)

puts posts.map(&:author)
puts posts.flat_map(&:comments).map(&:user)

Preload an scope

Some times you already have and scope that you want to reuse in a place where it can produce n+1 queries.

For example:

class Post < ApplicationRecord
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
  scope :popular, -> { where("votes_count > 5") }
end

class CommentVote < ApplicationRecord
  belongs_to :comment, counter_cache: :votes_count
end

If you ask for the popular comments of one post there is no problem.

post.comments.popular.count

But if you ask for the popular comments for each post on a list, you may have some n+1 queries.

posts = Post.limit(5)

posts.map do |post|
  # Here we are doing a new call to the database for each post
  post.comments.popular.size
end

Here the bad news is that you can’t really “just preload a scope”…

But the good news is that you can solve the problem by creating a new association for the scope and preload that association.

In the example, you can add a new has_many: :popular_comments, passing the scope on the association declaration, like this:

class Post < ApplicationRecord
  has_many :comments
  has_many :popular_comments, -> { popular }, class_name: "Comment"
end

class Comment < ApplicationRecord
  belongs_to :post
  scope :popular, -> { where("votes_count > 5") }
end

class CommentVote < ApplicationRecord
  belongs_to :comment, counter_cache: :votes_count
end

Now, in a similar way, you will be able to ask for the popular comments of one post without problems.

post.popular_comments.count

But now, if you need to ask for the popular comments for each post on a list, you can preload them to avoid the n+1 queries.

posts = Post.preload(:popular_comments).limit(5)

posts.map do |post|
  # Here we will use the preloaded comments. No n+1 queries!
  post.popular_comments.size
end

Preload the top 1 of an association

Imagine, that you have a model like this…

class Posts
  has_many :comments

  def latest_comment
    comments.order(:created_at).last
  end
end

When you are fetching a single post there is no problem…

post = Post.find(id)
puts post.latest_comment

But when you try to fetch a list of posts, the method seems to ignore the includes and runs a query for each post to get the latest_comment for each post.

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

One way of solving this problem is to instead order comments on the association definition and to later just ask for the last sorted comment.

class Post
  has_many :sorted_comments, -> { order(:created_at) }, class_name: "Comment"

  def latest_comment
    sorted_comments.last
  end
end

And now you will be able to preload the sorted_comments ask for the lastest_comment of each post without n+1 queries.

Post.includes(:sorted_comments).each do |post|
  puts post.latest_comment
end

The problem with this method is that you will fetch all the comments for every post in the list and sometimes it could be a problem.

But, you can avoid this problem!

You can use has_one association for the latest_comment, by providing a scope with just the latest_comment for each post.

class Post < ActiveRecord::Base
  has_many :comments
  has_one :latest_comment, -> { latest_comment_per_posts }, class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post

  def self.latest_comment_per_posts
    latest_comments_ids = select("max(id)").group(:post_id)
    where(id: latest_comments_ids)
  end
end

Here, in the first line of latest_comment_per_post we are building the sql to find the latest comment id for each post.

latest_comments_ids = select("max(id)").group(:post_id)

Something interesting is that the query will not be executed it will just pass the ActiveRecord::Relation to the where clause.

where(id: latest_comments_ids)

And now you can preload the has one association and fetch just the latest_comment for each post.

Post.includes(:latest_comment).all.each do |post|
  puts post.latest_comment.body
end

You can explore other ways of doing it on 5 ways to fix the latest-comment n+1 problem.

Preload the top n of an association

Sometimes you need to fetch just the top elements on a list, like:

The 3 most popular comments
The latest 3 comments
The latest review of a product
The 3 better prices for a product

For example, for the next model:

class Post < ApplicationRecord
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
end

If we want to fetch the latest 3 comments for a post, we can write something like this:

class Post < ApplicationRecord
  has_many :comments

  def latest_comments
    comments.order(id: :desc).limit(3)
  end
end

post.latest_comments

But as we saw on preload an scope, if we try to use it on a list we will get n+1 queries

Post.all.each do |post|
  # This will create a call for each post
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

And in this case, you can’t just move the scope to the association, because it will fetch only 3 comments in total, not for each post.

class Post < ApplicationRecord
  has_many :comments
  has_many :latest_comments, -> { order(id: :desc).limit(3) },
    class_name: "Comment"
end

Post.includes(:latest_comments).limit(10).map do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

On the the sql produced on the preload, you can see that we are just fetching 3 comments in total.

SELECT "posts".* FROM "posts" LIMIT 10

SELECT "comments".*
  FROM "comments"
  WHERE "comments"."post_id" IN (...)
  ORDER BY "comments"."id" DESC
  LIMIT 3

One way to solve the problem could be to, keep just the order on the association declaration, and do the selection in ruby, like this:

class Post < ApplicationRecord
  has_many :comments
  has_many :sorted_comments, -> { order(id: :desc) },
    class_name: "Comment"
end

class Comment < ApplicationRecord
  belongs_to :post
end

Post.includes(:sorted_comments).limit(10).map do |post|
  post.sorted_comments.to_a.first(3).each do |comment|
    puts comment.body
  end
end

But if you have many comments per post it can be a problem. Because it will load them all.

You can check the Benchmarks for the fixes to the latest-comment n+1 problem to get more specific on what is “many comments”.

One way to solve this problem is to use a lateral join. There are more, but you can start with this.

class Post < ApplicationRecord
  has_many :comments
  has_many :latest_comments, -> { latest_comments_per_post },
    class_name: "Comment"
end

class Comment < ApplicationRecord
  belongs_to :post

  def self.latest_comments_per_post
    latest_comments = Post
      .select("latest_comments.*")
      .joins(<<-SQL)
        JOIN LATERAL (
          SELECT * FROM comments
          WHERE post_id = posts.id
          ORDER BY id DESC LIMIT 3
        ) AS latest_comments ON TRUE
      SQL

    from(latest_comments, "comments").order(id: :desc)
  end
end

Post.includes(:latest_comments).each do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

On the method latest_comments_per_posts we are fetching just the latest three comments per posts. Because it will apply the limit “inside” the join.

def self.latest_comments_per_post
  latest_comments = Post
     # Here we select just the "comments"
    .select("latest_comments.*")

    # Here we are doing the limit inside the join,
    # it will apply it for the comments of each post.
    .joins(<<-SQL)
      JOIN LATERAL (
        SELECT * FROM comments
        WHERE post_id = posts.id
        ORDER BY id DESC LIMIT 3
      ) AS latest_comments ON TRUE
    SQL

  # Here we use the returned "latest_comments" and
  # we name them like "comments". It will tell rails
  # to instantiate Comment objects
  from(latest_comments, "comments").order(id: :desc)
end

You can explore other ways of doing it on 5 ways to fetch the latest-N-of-each record on Rails.

Simplify preloading with has_many through

Preloading nested associations is not that hard, but sometimes could be unnecessary for certain actions.

For example, with the next model:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
end

Imagine that you want to fetch the people that have a vote on a comment in a post, the “comment voters”.

To implement it, you can have a method in the post to do that…

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments

  def comment_voters
    comments.preload(votes: :voter)
      .flat_map(:votes)
      .flat_map(:voter)
      .uniq
  end
end

And it will correctly preload the votes and voters to compute the result

puts post.comment_voters

But if you try to do it for a list of post, it will execute n+1 queries, and also you will be loading a bunch of data that you don’t need.

You can try to do a “better query” to avoid preloading all that data, but making it work in a list will be very hard.

Fortunately you can use a has_many through: associations to do this operation in a much simpler way.

You can add a has_many :voters to Comment and a has_many :comment_voters to Post.

Like this:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
  has_many :comment_voters, -> { distinct }, through: :comments, source: :voters
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
  has_many :voters, through: :votes
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
end

This will correctly compute the data in just one query

puts post.comment_voters

But, you will also be able to preload the comment_voters just as any other association.

Post.preload(:comment_voters).all.each do |post|
  puts post.comment_voters.to_a
end

Preload object

Sometimes using using the default preloading mechanisms from rails, preload, includes and eager_load is not possible… or maybe it is, but you just can’t figure it out how to do it.

Maybe you know how to represent the association with a has_many :through, but you need something faster, or to use less memory.

In that kind of situations, one thing that you can do is to write a custom object to represent that preload.

For example, with the example from the last section:

class Post < ApplicationRecord
  belongs_to :author, class_name: "User"
  has_many :comments
  has_many :comment_voters, -> { distinct }, through: :comments, source: :voters
end

class Comment < ApplicationRecord
  belongs_to :post
  belongs_to :user
  has_many :votes, class_name: "CommentVote"
  has_many :voters, through: :votes
end

class CommentVote < ApplicationRecord
  belongs_to :voter, class_name: "User"
  belongs_to :comment
end

class User < ApplicationRecord
end

You know that this will correctly compute the data in just one query

post = Post.first
post.comment_voters

And, that you will also be able to preload the comment_voters just as any other association.

Post.preload(:comment_voters).all.each do |post|
  puts post.comment_voters.to_a
end

But…

If you see the executed queries, it could be not as performant as you need…

irb(main):055:0> Post.preload(:comment_voters).to_a.count

Post Load (0.7ms)  SELECT "posts".* FROM "posts"

Comment Load (3.5ms)  SELECT "comments".* FROM "comments" WHERE
"comments"."post_id" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)  [["post_id", 1], ["post_id", 2], ["post_id", 3], ["post_id", 4],
["post_id", 5], ["post_id", 6], ["post_id", 7], ["post_id", 8], ["post_id", 9],
["post_id", 10], ["post_id", 11], ["post_id", 12], ["post_id", 13], ["post_id",
14], ["post_id", 15], ["post_id", 16]...

CommentVote Load (18.3ms)  SELECT "comment_votes".* FROM "comment_votes" WHERE
"comment_votes"."comment_id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74,
75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94,
95, 96, 97, 98, 99, 100...

User Load (0.2ms)  SELECT DISTINCT "users".* FROM "users" WHERE "users"."id" IN
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)  [["id", 8], ["id", 5], ["id", 1], ["id", 6],
["id", 4], ["id", 7], ["id", 3], ["id", 10], ["id", 2], ["id", 9]]

Maybe you can try to eager_load, but maybe it could not be what you need either…

irb(main):059:0> posts = Post.eager_load(:comment_voters).to_a

SQL (47.0ms)  SELECT "posts"."id" AS t0_r0, "posts"."author_id" AS t0_r1,
"posts"."title" AS t0_r2, "posts"."body" AS t0_r3, "posts"."created_at" AS
t0_r4, "posts"."updated_at" AS t0_r5, "users"."id" AS t1_r0, "users"."name" AS
t1_r1, "users"."created_at" AS t1_r2, "users"."updated_at" AS t1_r3 FROM
"posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" LEFT
OUTER JOIN "comment_votes" ON "comment_votes"."comment_id" = "comments"."id"
LEFT OUTER JOIN "users" ON "users"."id" = "comment_votes"."voter_id"

Trying a custom query

Maybe what you want is to write a custom query, because you don’t want to fetch data that you don’t need.

Let’s imagine that what you need is to fetch just the users with voted comments on a collection of posts. Without preloading anything else, like this:

posts = Post.limit(30)
comment_voters = User
  .select("users.*, comments.post_id") # selects the user attributes and the post_id of the comment.
  .joins(comment_votes: [:comment])
  .distinct
  .where(comments: { post_id: posts })
  .to_a

User Load (16.5ms)  SELECT DISTINCT users.*, comments.post_id FROM "users"
INNER JOIN "comment_votes" ON "comment_votes"."voter_id" = "users"."id" INNER
JOIN "comments" ON "comments"."id" = "comment_votes"."comment_id" WHERE
"comments"."post_id" IN (SELECT "posts"."id" FROM "posts")

You can then group this voters by post_id, and then in the view you can pick the voters for each post.

posts = Post.limit(30)
comment_voters = User
  .select("users.*, comments.post_id") # selects the user attributes and the post_id of the comment.
  .joins(comment_votes: [:comment])
  .where(comments: { post_id: posts })
  .distinct
  .group_by(&:post_id)

posts.each do |post|
  comment_voters[post.id].map(&:name)
end

That’s ok, but…

What if you don’t want to expose that code in your controller or view?… What if instead we move that to a model object?… Where should you put that code?

A preload object

There are many options, but one of them, is to create a new object to help you with two things:

Fetch the voters for the posts in one call to the database
Return the voters for a given post

For this example you can call the object Post::CommentVotersPreload and use it like this:

posts = Post.limit(30)
comment_voters = Post::CommentVotersPreload.new(posts)

posts.each do |post|
  comment_voters.for_post(post).map(&:name)
end

And the implementation could look something like this:

class Post::CommentVotersPreload
  def initialize(posts)
    @posts = posts
  end

  def for_post(post)
    comment_voters[post.id]
  end

  private

  def comment_voters
    @comment_voters ||= fetch_records.group_by(&:post_id)
  end

  def fetch_records
    User.select("users.*, comments.post_id")
      .joins(comment_votes: [:comment])
      .where(comments: { post_id: @posts })
      .distinct
  end
end

You can use this kind of objects in many different problems.

What you will need is:

A fetch_records method to fetch the records you want to preload in one call to the database (or at least not n+1 queries).
A find_for method to return the records related to each associated object.

If you need more examples to understand the idea, you can check this other examples:

When should you use it?

The answer to this question is not that simple (at least for me).

What I could recommend you is to first try to solve the problem in a “railsy” way, using the mechanisms, provided by rails. Normally those mechanisms will compose better with other parts of the framework.

But if you have already tried what rails provides, and you can’t find a way to do what you need to do… or you need something more efficient… or you just think that the final code is better…

Then go for it and use a “preload object”, just try to have a reason.

Final notes

You made to the end! Congrats!

Let’s recap what you can do now…

Preload direct associations
Preload nested associations
“Preload an scope”… you know that you need an association
Preload the top n of an association… with a lateral join an maybe with other methods
Simplify a preloading with a has_many through association
Use a preload object to represent a preload

I hope that you can feel that now you are a better rails developer!

I hope you have enjoyed this guide!

If you have some feedback on how this guide can help you better, please write a comment =)

What is an n+1 queries problem?

2021-07-27T12:56:00Z

An n+1 queries problem means that a query is executed for every result of a previous query.

For example, with this records…

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

If you execute the next code…

Post.all.each do |post|
  post.comments.each do |comment|
    puts comment.body
  end
end

You will execute…

1 query to fetch the posts (Post.all)
And n queries to fetch the comments of each post (post.comments)

If there are…

2 posts, it will execute 3 queries (2 + 1)
5 posts, it will execute 6 queries (5 + 1)
1000 posts, it will execute 1001 queries (1000 + 1)

Tool to run the examples for the 5 ways to fetch the latest-n-of-each record

2021-07-22T12:56:00Z

A common cause of n+1 queries is fetching the “latest-N-of-each” record on a list of records.

Some examples of this problem are trying to get…

The latest comments for each post
The latest payments for each customer
The latest review for each product
The better prices for each product
…

I have one post named 5 ways to fetch the latest-N-of-each record on Rails where I share different ways to solve this problem. Here I want to share with you the code that I used in the post.

You can use it to run the examples, and play with the code and seed values, to pick the right solution for you current case.

How to run the examples

1 - Install the dependencies with bundle install.

2 - Database setup - run the command:

ruby db/setup.rb

3 - Run the examples with ruby fixes_examples/<file name>. For example:

ruby fixes_examples/01_fetch_all_and_select_for_each.rb

4 - Change the posts and comments count on db/seeds.rb you can change Seeds.run(posts_count: 10, comments_count: 10) and re-run ruby db/setup.rb to test different scenarios.

Get the code to run the examples

Subscribe, and get the code to run each of the examples in the post. You will receive a zip with the code.

How to test that an email was sent on a unit/micro test

2021-07-12T00:28:00Z

There are different practices, that I know, to test that an email was sent in Rails…

Check for the ActionMailer::Base.deliveries
Spy your own sender objects
Use doubles for your mailer objects

All of them have pros and cons and will help you test different things.

That is why I think that is good to understand all of them because it will help you decide which method is better case by case.

1. Check for the ActionMailer::Base.deliveries

It will be similar to what you will do on a feature/capybara spec.

You will check the ActionMailer::Base.deliveries to see if the email that you want is there.

RSpec.describe Event, "#create_invitation" do
  it "sends the invitation email" do
    event = create :event

    event.create_invitation(name: "Mary", email: "mary@example.com")

    mail = find_mail_to("mary@example.com")

    expect(mail.subject).to eq "You have been invited to #{event.name}"
  end

  def find_mail_to(email)
    ActionMailer::Base.deliveries.find { |mail|
      mail.to.include?(email)
    }
  end
end

2. Spy your own sender objects

Using rspec, you can expect that an object will receive the right method that will actually deliver the mail.

RSpec.describe Event, "#send_invitation_emails" do
  it "sends the email for all invitations" do
    event = create :event
    invitation1 = create :invitation, event: event
    invitation2 = create :invitation, event: event

    expect(invitation1).to receive(:send_email)
    expect(invitation2).to receive(:send_email)

    event.send_invitation_emails
  end
end

3. Use doubles for your mailer objects

Using rspec, you can allow your mailer to receive new with the right attributes and then return an object that will receive the method that actually delivers the mail.

RSpec.describe Invitation, "#send_email" do
  it "sends the new_invitation email" do
    invitation = build :invitation

    mail = double("Mail")

    allow(EventMailer).to receive(:new_invitation).with(invitation).and_return(mail)

    expect(mail).to receive(:deliver_later)

    invitation.send_email
  end
end

What method should you use?

As you can see not all tests will test exactly the same thing. And most of the time you won’t need all of them.

If you are struggling deciding which test to write, I recommend you to start with the feature/capybara spec.

Then if you fill that you need to test something at the unit/micro level, you can try this methods in order, but just moving to the next method only if you really feel that is necesary.

So, you can start with the first method, then if you need more confidence you can try the second, and finally the third method if you really think that is necessary.

How to test that an email was sent on a system spec

2021-07-02T00:28:00Z

If you want to test that an email was sent on a system spec in rails…

What you can do is to check the ActionMailer::Base.deliveries to see if the email that you want is there.

You can look for the mail with something like this:

mail = ActionMailer::Base.deliveries.find { |mail|
  mail.to.incude? "mary@example.com"
}

Maybe you can write a helper method to make the code a little more readable:

def find_mail_to(email)
  ActionMailer::Base.deliveries.find { |mail| mail.to.include?(email) }
end

And you can check if the subject is right with something like this:

expect(mail.subject).to eq "Super subject!"

A full example

Imagine that you are building an app to manage events, and you are writing a feature to invite guests to an event via email.

So, in your system spec you want to check that when you create an invitation the system will send the right email to the invited guest.

Then you can write something like this in your spec:

RSpec.describe "Event owner invites guest" do
  scenario "successfully" do
    event = create :event

    login_as event.owner

    visit event_path(event)

    create_invitation(name: "Mary", email: "mary@example.com")

    mail = find_mail_to "mary@example.com"

    expect(mail.subject).to eq "You have been invited to #{event.name}"
  end

  def create_invitation(name:, email:)
    click "Invite guest"
    fill_in "Name", with: name
    fill_in "Email", with: email
    click_on "Create invitation"
  end

  def find_mail_to(email)
    ActionMailer::Base.deliveries.find { |mail| mail.to.include?(email) }
  end
end

Guide to name scopes based on column types

2021-06-29T00:28:00Z

As you may now there are two hard things in computer science and one of them is naming things.

And although naming things is an art, is not bad to have a little guide on how to do it.

That’s why I want to share with you a guide on how to name scopes that depend on a single column based on its type.

This suggestion is based on a very interesting gem called type_scopes, that I found on reddit.

I think that even if you don’t use the gem you can benefit from the names that it is proposing. Maybe it won’t fit all cases but I think that it could make our work a little easier.

Date

Record.received_to("2017-09-06")
# => where("received_on <= '2017-09-06'")

Record.received_from("2017-09-06")
# => where("received_on >= '2017-09-06'")

Record.received_after("2017-09-06")
# => where("received_on > '2017-09-06'")

Record.received_before("2017-09-06")
# => where("received_on < '2017-09-06'")

Record.received_between("2017-09-06", "2017-09-07")
# => where("received_on BETWEEN '2017-09-06' AND '2017-09-07'")

Record.received_not_between("2017-09-06", "2017-09-07")
# => where("received_on NOT BETWEEN '2017-09-06' AND '2017-09-07'")

Record.received_within("2017-09-06", "2017-09-07")
# => where("received_on > '2017-09-06' AND received_on < '2017-09-07'")

Record.received_not_within("2017-09-06", "2017-09-07")
# => where("received_on <= '2017-09-06' OR received_on >= '2017-09-07'")

Datetime

Record.completed_to("2017-09-06")
# => where("completed_at <= '2017-09-06'")

Record.completed_from("2017-09-06")
# => where("completed_at >= '2017-09-06'")

Record.completed_after("2017-09-06")
# => where("completed_at > '2017-09-06'")

Record.completed_before("2017-09-06")
# => where("completed_at < '2017-09-06'")

Record.completed_between("2017-09-06", "2017-09-07")
# => where("completed_at BETWEEN '2017-09-06' AND '2017-09-07'")

Record.completed_not_between("2017-09-06", "2017-09-07")
# => where("completed_at NOT BETWEEN '2017-09-06' AND '2017-09-07'")

Record.completed_within("2017-09-06", "2017-09-07")
# => where("completed_at > '2017-09-06' AND completed_at < '2017-09-07'")

Record.completed_not_within("2017-09-06", "2017-09-07")
# => where("completed_at <= '2017-09-06' OR completed_at >= '2017-09-07'")

Numeric

Record.price_to(100)
# => where("price <= 100")

Record.price_from(100)
# => where("price >= 100")

Record.price_above(100)
# => where("price > 100")

Record.price_below(100)
# => where("price < 100")

Record.price_between(100, 200)
# => where("price BETWEEN 100 AND 200")

Record.price_not_between(100, 200)
# => where("price NOT BETWEEN 100 AND 200")

Record.price_within(100, 200)
# => where("price > 100 AND price < 200")

Record.price_not_within(100, 200)
# => where("price <= 100 OR price >= 200")

String

Record.name_contains("foo")
# => where("name LIKE '%foo%'")

Record.name_contains("foo", sensitive: false)
# => where("name ILIKE '%foo%'")

Record.name_starts_with("foo")
# => where("name LIKE 'foo%'")

Record.name_starts_with("foo", sensitive: false)
# => where("name ILIKE 'foo%'")

Record.name_ends_with("foo")
# => where("name LIKE '%foo'")

Record.name_ends_with("foo", sensitive: false)
# => where("name ILIKE '%foo'")

Record.name_like("%foo%")
# => where("name LIKE '%foo%'")

Record.name_ilike("%foo%")
# => where("name ILIKE '%foo%'")

Boolean

Record.completed
# => where("completed = true")

Record.not_completed
# => where("completed = false")

Record.is_native
# => where("is_native = true")

Record.is_not_native
# => where("is_native = false")

Record.has_invitation
# => where("has_invitation = true")

Record.has_not_invitation
# => where("has_invitation = false")

Record.was_terminated
# => where("was_terminated = true")

Record.was_not_terminated
# => where("was_terminated = false")

List of alternatives to heroku

2021-06-22T00:28:00Z

I like heroku, it is very easy to use, but sometimes you just can’t use it…

Maybe it is too expensive for your project. Maybe you need something that it does not provide, like an integrated “S3-storage”. Maybe for some reason you are forced to run on AWS. Or maybe you prefer just a layer on top of the hosting provider.

Here you can find a list of tools/services with similar characteristics that you can check if you need find an alternative.

AWS elastic beanstalk

Description:

“AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.”

Price: There is no additional charge for AWS Elastic Beanstalk. You pay for AWS resources. You can see more on their pricing page.

Cloud 66

Description:

“Ops Tools for Devs. Our products are like your in-house DevOps team. They take care of your uptime, scaling, and security. Just like your own team would, but with the experience of millions of deployments per day.”

Price: Price: 14.99 / server per month. You can see more on their pricing page

Digital Ocean App Platform

Description:

“Build, deploy, and scale apps quickly using a simple, fully managed solution. We’ll handle the infrastructure, app runtimes and dependencies, so that you can push code to production in just a few clicks.”

Price: Starts at $5/mo

Dokku

Description:

“The smallest PaaS implementation you’ve ever seen. Dokku helps you build and manage the lifecycle of applications”

Price: Free

Fly.io

Description:

“Deploy App Servers Close to Your Users. Run your full stack apps (and databases!) all over the world. No ops required.”

Price: Starts free. You can see more on their pricing page

Hatchbox.io

Description:

“Build, deploy, and manage Rails apps. Ruby on Rails apps on your own hosting provider in minutes and without hassle.”

Price: Starts at $29 per month, deploy as many apps as you want.

Render

Description:

“Render is a unified cloud to build and run all your apps and websites with free SSL, a global CDN, private networks and auto deploys from Git.”

Price: From $7/mo

Guide to justify the heroku's price to your boss or clients

2021-06-14T00:28:00Z

Maybe your are thinking that you should just use Heroku and move on with your life. Do something that provides value to the customer instead of host configurations.

But is not always that easy, because if you just compare the cost of GB of Ram or CPUs, it does not look that well… $50 for 1gb is barely enough and maybe it won’t let you sleep at night… But then $250 for 2.5gb or $500 for 14gb could be more than enough.

So you would need to ask your boss or clients something between $50-500/month, and maybe it could look like too much.

Maybe Heroku is actually really expensive for your project, but maybe not, here I want to share with you some things that you could do to justify the price to your boss or client if you think that heroku (or another PaSS) is the right choice for your project.

Identify the things that you need from heroku

Remember the things that heroku is doing for you, that your project needs, and use them as a tool to justify the price to your boss or client.

Here a list some of the features of heroku but there are more…

Backup system
Git push based, deployment system
Automatic OS updates
Review apps
Semi-automatic scaling
Addonds
Environment variables system
High compliance requirements
Process monitoring system
Access management
Continuos integration system
Github integration

Remember what they are actually paying for developers

If you think that they are already paying between $20 and $200 per hour, maybe increasing your price a little to cover other $50-$500 per month, could be possible if you list all the things that they could have for that money (take them from the list above).

Remember that devops is a thing

If your service includes keeping the application up, charging for that is ok.

Think how much this could cost if you outsource it…

How many hours from you?
How many hours from the devops person/team?
How much in plain hosting?

You could use that number to justify the increase of your price.

Remember that devops can go wrong

If you are not a devops pro, or even if you hire a team of senior devops, things can go wrong and its possible that even after spending a lot of money on the migration from heroku, the project must be moved to heroku again.

List of questions to check if heroku is really expensive for your project

2021-06-08T00:28:00Z

For most of us, heroku is a really useful tool because it does so much for us, but in some cases it could look like it is really expensive…

… Is it really expensive? Does it worth the time it saves on tooling, maintenance, devops, etc? Would you be able to replace it with AWS or other provider?

I really don’t know. I think it really depends on you, your team and your current project.

But if you are reading this article is because maybe you have a project where you are wondering if it really worth the price.

So, here I want to share with you a list of questions, to help you decide if heroku is reall expensive for your project or not.

What do you need?

Do you even care about all the features in heroku or you just want something to play?
How important is a backup system?
How important is to just git push?
How important are automatic OS updates?
How important are automatic security updates?
How important are review apps?
How important are the semi-automatic scaling?
How many addons do you depend on that will be hard to setup with other cloud provider?
Do you need high complience level requirements that heroku already can give you?

What do you know?

Do you know how to run your app in another cloud provider?
Do you know how to do a deployment setup for another cloud provider?
Do you know how to setup a backup system in another cloud provider?
Do you know how to do and keep track of OS updates?
Do you know how to do and keep track of security updates?
Do you know how setup a review apps system?
Do you know how setup a resource scaling system?

Current bill

Is your current bill larger than the bill of the other provider plus the salary cost of at least one senior devops?
Are you aware that even hire two senior devops, for the migration from heroku can end in a fail as a project?
Do you want to do all the required work to avoid the bill?
Have you tried to pitch to you client, boss or the person who pays the bills how much does a migration will cost? (the other provider bill, plus the salary cost of at least one senior devops, even if this person if you)

A rule of thumb

Normally Heroku will be more expensive than doing the work yourself on other provider, but it will almost always going to be cheaper than hiring a devops person to do on other provider what Heroku does for you.

You can think of heroku as “hosting + devops”.

So, if you know some devops and are comfortable handling an emergency without a devops person, AWS or other providers will probably be cheaper than Heroku.

If you would need to hire someone to handle devops or it will take much from your own time and you don’t want it, I think that you are better using Heroku even if it looks “expensive”.

Extracting domain knowledge from a query method

2021-05-27T00:28:00Z

Imagine that you need to query for the payments received the “week before” of a given date.

How would you expose this feature in your model?

In this article I want to share some ways of doing it, by telling the story of what we did in a similar case…

Note: I am changing the names a little but the problem was mostly the same

Why do we need that query?

We needed that query because we wanted to send a notification to all users that received a payment in the last week. That notification will be sent each week.

Our first approach…

The first thing that we tried was to expose a class method/scope in the Payment record, like this…

class Payment
  def self.received_this_week
    #...
  end
end

We wrote some tests…

RSpec.describe Payment do
  describe ".received_a_week_before" do
    example "payment received on last second" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 13, 23, 59)
      result = described_class.received_a_week_before(Date.new(2020, 2, 14))
      expect(result).to eq([payment])
    end

    example "payment received one second late" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 14, 00, 00)
      result = described_class.received_a_week_before(Date.new(2020, 2, 14))
      expect(result).to be_empty
    end

    example "payment received in the first second" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 7, 00, 00)
      result = described_class.received_a_week_before(Date.new(2020, 2, 14))
      expect(result).to eq([payment])
    end

    example "payment received on previous week" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 6, 23, 59)
      result = described_class.received_a_week_before(Date.new(2020, 2, 14))
      expect(result).to be_empty
    end
  end
end

To end with an implementation like this…

class Payment < ApplicationRecord
  def self.received_this_week(date = Date.current)
    range = (date - 7.days).beginning_of_day..(date - 1.day).end_of_day
    where(received_at: range)
  end
end

Note: if you want to know about how to build the range, you can see Building a DateTime range to query for the records created a “week before”

This was ok, but…

This was ok and it was working, but the name was not really that expressive, at least for us (on that moment…).

As I said, we were implementing this method because we wanted to send a notification that was going to be sent each week.

Maybe we could have changed the name to something like Payment.for_weekly_notification but we started to think that this behavior, should not be on the Payment class, instead it could belong to a more specific object for this specific use case…

A second a approach…

As we were working to create a “weekly notification with the received payments”…

… We thought, that we could move the construction of the range to a new object and then ask Payment for the payments received between that range.

So we end with something like this…

range = Payment::WeeklyNotificationRange.for_date(Date.current)
Payment.received_between(range)

And later we add a for_today method…

range = Payment::WeeklyNotificationRange.for_today
Payment.received_between(range)

This is more verbose, but at least for us, the intent was a little more clear.

So we wrote some tests for the WeeklyNotificationRange…

RSpec.describe Payment::WeeklyNotificationRange do
  describe ".for_date" do
    example "on friday" do
      result = described_class.for_date(Date.new(2020, 02, 14))
      expect(result).to eq(Time.zone.local(2020, 02, 7).beginning_of_day..Time.zone.local(2020, 02, 13).end_of_day)
    end

    example "on thursday" do
      result = described_class.for_date(Date.new(2020, 02, 13))
      expect(result).to eq(Time.zone.local(2020, 02, 6).beginning_of_day..Time.zone.local(2020, 02, 12).end_of_day)
    end
  end

  describe ".for_today" do
    it "returns range for current day" do
      result = described_class.for_today
      expect(result).to eq(described_class.for_date(Date.current))
    end
  end
end

Extracting this method help us to express more clearly in the specs how the range will change if we ask for a different date. I think that this was a win!

Then we wrote the implementation (Well, we just moved it)…

class Payment::WeeklyNotificationRange
  def self.for_date(date)
    (date - 7.days).beginning_of_day..(date - 1.day).end_of_day
  end

  def self.for_today
    for_date(Date.current)
  end
end

We changed the Payment specs…

RSpec.describe Payment do
  describe ".received_between" do
    let(:range) do
      Time.zone.local(2020, 02, 7).beginning_of_day..Time.zone.local(2020, 02, 13).end_of_day
    end

    example "payment received on last second" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 13, 23, 59)
      result = described_class.received_between(range)
      expect(result).to eq([payment])
    end

    example "payment received one second late" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 14, 00, 00)
      result = described_class.received_between(range)
      expect(result).to be_empty
    end

    example "payment received in the first second" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 7, 00, 00)
      result = described_class.received_between(range)
      expect(result).to eq([payment])
    end

    example "payment received on previous week" do
      payment = create :project_payment, received_at: Time.zone.local(2020, 02, 6, 23, 59)
      result = described_class.received_between(range)
      expect(result).to be_empty
    end
  end
end

And the implementation…

class Payment < ApplicationRecord
  def self.received_between(range)
    where(received_at: range)
  end
end

We stopped here…

We decided to let the code like this, because we thought it was clear enough…

We used this code to run a rake task that sends an email to each user that received a payment in the last week…

if Date.current.friday?
  range = Payment::WeeklyNotificationRange.for_today

  Payment.received_between(range).extract_users.find_each do |user|
    UserMailer.received_payments_notification(user).deliver_later
  end
end

And in the mailer we did something like this…

class UserMailer < ApplicationMailer
  def received_payments_notification(user)
    range = Payment::WeeklyNotificationRange.for_today
    @payments = Payment.for_user(user).received_between(range)
    # mail(...)
  end
end

But we could have abstracted a little more…

Again, for us this was enough, but…

I want to share with you how you can encapsulate something like this a little more, because maybe it could be useful in some circumstances…

We can add a Payment::WeeklyNotification to handle the tasks related with this use case…

For example in our rake task we can do something like this…

if Date.current.friday?
  notification = Payment::WeeklyNotification.new

  notification.users.find_each do |user|
    UserMailer.received_payments_notification(user).deliver_later
  end
end

Or…

if Date.current.friday?
  notification = Payment::WeeklyNotification.new

  notification.users.find_each do |user|
    notification.email_for(user).deliver_later
  end
end

Or maybe just…

if Date.current.friday?
  Payment::WeeklyNotification.new.notify_users
end

And in the mailer we can do something like this…

class UserMailer < ApplicationMailer
  def received_payments_notification(user)
    @payments = Payment::WeeklyNotification.new.payments_for(user)
    # mail(...)
  end
end

And move the logic to that object…

class WeeklyNotification
  def users
    Payment.received_between(date_range).extract_users
  end

  def notify_users
    users.find_each do |user|
      email_for(user).deliver_later
    end
  end

  def email_for(user)
    UserMailer.received_payments_notification(user)
  end

  def payments_for(user)
    Payment.for_user(user).received_between(date_range)
  end

  private

  def date_range
    Payment::WeeklyNotificationRange.for_today
  end
end

And, of course add some tests… Or move them…

But you should stop somewhere…

But you should stop somewhere, and there are trade-offs in each decision…

No solution is perfect… Each abstraction requires maintenance (tests, documentation, etc…), but if you avoid them the intent of the code could be less clear.

How could you know where to stop?

Is really hard to know, but I like the BeckDesignRules …

Kent Beck came up with his four rules of simple design while he was developing ExtremeProgramming in the late 1990’s. And Martin Fowler express them like this:

Passes the tests
Reveals intention
No duplication
Fewest elements

The rules are in priority order, so “passes the tests” takes priority over “reveals intention”.

You can use this rules as a guide =)

Building a DateTime range to query for the records created a "week before"

2021-05-06T00:28:00Z

Imagine that you need to query for the records created in the “week before” of a given date.

How would you do it?

Example

For this example a “week before” means that…

If the given date is the Friday 2021-04-30 it will return the records created between the begining of the previous Friday 2021-04-23 and the last second of the Thursday 2021-04-29.

Asking to ActiveRecord

In rails, with ActiveRecord, you can ask for the records created between a range with…

Record.where(created_at: range)

This will build a BETWEEN query with the range you have provided, but how do you build the range?…

Let’s see!

Building the range

Let’s start with the Friday 2021-04-30 from the example…

date = Date.new(2021, 4, 30)
# => Fri, 30 Apr 2021

If we substract 7 to it, it will return the previous Friday…

date - 7
# => Fri, 23 Apr 2021

It will do the same if we substract 7.days

date - 7.days
# => Fri, 23 Apr 2021

What happen if we substract 7.days.ago?…

It will return an error because 7.days.ago is not a number…

date - 7.days.ago
# Traceback (most recent call last):
#         1: from (irb):4
# TypeError (expected numeric)

It is actually a Datetime…

7.days.ago
# => Fri, 23 Apr 2021 10:22:54 CDT -05:00

We can convert it to a date with to_date…

7.days.ago.to_date
# => Fri, 23 Apr 2021

But let’s return to the range…

Now to get the Thursday 2021-04-29 we need to substract 1.day to our date

date - 1.day
# => Thu, 29 Apr 2021

So, to build the desired range we can do…

range = (date - 7.days)..(date - 1.day)
# => Fri, 23 Apr 2021..Thu, 29 Apr 2021

Or (without days and day)…

range = (date - 7)..(date - 1)
# => Fri, 23 Apr 2021..Thu, 29 Apr 2021

That will return a range of Date objects…

range.to_a
# => [Fri, 23 Apr 2021, Sat, 24 Apr 2021, Sun, 25 Apr 2021, Mon, 26 Apr 2021, Tue, 27 Apr 2021, Wed, 28 Apr 2021, Thu, 29 Apr 2021, Fri, 30 Apr 2021]

But normally our datetime fields store the information for utc. So, we should not query directly with the dates, because it won’t match the expected timezone.

We need to build the range aware of the time zone that we need and then let rails transform the range to utc.

To do that we can use the rails methods beginning_of_day and end_of_day to convert our date in to a datetime aware of the time zone.

range = ((date - 7).beginning_of_day)..((date - 1.day).end_of_day)
# => Fri, 23 Apr 2021 00:00:00 CDT -05:00..Fri, 30 Apr 2021 23:59:59 CDT -05:00

And now that we have the range that we want we can pass it to the rails where method…

Record.where(created_at: range)

Try this code in the console

To understand this code better, try to run each example in the rails console and to test the things that maybe are not so clear at this moment.

How to read the noCertificado from a CSD for a mexican CFDI

2021-05-05T00:28:00Z

This was new for me, I was looking how to get the noCertificado from a CSD (Certificado de Sello Digital) and I found it is the serial of the certificate but in base 2.

In ruby you can get it with a OpenSSL::X509::Certificate

raw = File.read("my_cert.cer")
cert = OpenSSL::X509::Certificate.new(raw)

If you ask the serial you will get an OpenSSL::BN

cert.serial # => an OpenSSL::BN

And then you can ask the for the to_s of that object. By default it will return it with base 10

cert.serial.to_s == cert.serial.to_s(10) # => true

…but you can ask for it in base 2, this is the noCertificado =)

cert.serial.to_s(2) # => "noCertificato" value...

How to test a PDF download with capybara

2021-05-03T00:28:00Z

Do you want to test a PDF download with capybara, but don’t know how to do it?

Here I will how you can doing it are using the default driver (not the javascript driver).

What do you want to test?

The first thing that you are going to need, is knowing what do you want to test, here I will show you how you can test…

That you are downloding the expected content
with the right filename
the right content_type
and that disposition is the one that you actually want

… but maybe some of them are not important for your specific use case, so you can change the code to fit your needs.

How to test if the content is right

You can compare the page.body with the content of the file that you want to send.

For example if you are using carrierwave and have a Receipt record like this…

class Receipt < ApplicationRecord
  belongs_to :user
  mount_uploader :pdf, PdfUploader
end

You can send the pdf like this…

class ReceiptsController < ApplicationController
  def show
    receipt = Receipt.find(params[:id])

    send_data receipt.pdf.read,
      type: receipt.pdf.content_type,
      filename: "Receipt-#{receipt.id}.pdf",
      disposition: "inline"
  end
end

You can write a test like this…

RSpec.describe "User downloads receipt pdf" do
  scenario "download from the receipts page" do
    user = create :user

    create :receipt, user: user

    login_as user, scope: :user

    visit receipts_path

    click_on "Download"

    expect(page.body).to eq receipt.pdf.read
  end
end

How to test if the filename is right

To test the filename you can check the Content-Disposition header, because it has the form…

#{disposition}; filename=#{filename}

RSpec.describe "User downloads receipt pdf" do
  scenario "download from the receipts page" do
    user = create :user

    receipt = create :receipt, user: user

    login_as user, scope: :user

    visit receipts_path

    click_on "Download"

    expect(page.response_headers["Content-Disposition"]).to match "Receipt-#{receipt.id}.pdf"
  end
end

How to test if the content type is right

To test content_type you can check the Content-Type header, like this…

RSpec.describe "User downloads receipt pdf" do
  scenario "download from the receipts page" do
    user = create :user

    receipt = create :receipt, user: user

    login_as user, scope: :user

    visit receipts_path

    click_on "Download"

    expect(page.response_headers["Content-Type"]).to eq "application/pdf"
  end
end

How to test if the disposition is right

To test the disposition you also can check the Content-Disposition header, because it has the form…

#{disposition}; filename=#{filename}

RSpec.describe "User downloads receipt pdf" do
  scenario "download from the receipts page" do
    user = create :user

    receipt = create :receipt, user: user

    login_as user, scope: :user

    visit receipts_path

    click_on "Download"

    expect(page.response_headers["Content-Disposition"]).to match "inline"
  end
end

A helper to test all together

If you want to test the four things together (or some of them), you can write a helper function to express that you want a download with some characteristics. You can do something like this…

RSpec.describe "User downloads receipt pdf" do
  scenario "download from the receipts page" do
    user = create :user

    receipt = create :receipt, user: user

    login_as user, scope: :user

    visit receipts_path

    click_on "Download"

    expect_download(
      content: receipt.pdf.read,
      filename: "Receipt-#{receipt.id}.pdf",
      content_type: "application/pdf",
      disposition: "inline"
    )
  end

  def expect_download(content:, filename:, content_type:, disposition:)
    expect(page.response_headers['Content-Type']).to eq content_type
    expect(page.response_headers['Content-Disposition']).to eq "#{disposition}; filename=\"#{filename}\""
    expect(page.body).to eq content
  end
end

… and that’s all for now. I hope it helps =)

A simple way of testing a CSV download with capybara

2021-04-20T00:28:00Z

Do you need/want to test a CSV download with capybara, but don’t know how to do it?

Here I will show you a simple way of doing it if you are using the default driver.

CSV example

Imagine that in your feature you are exporting a csv like this one…

User, Email, Amount
Juan, juan@example.com, $10.00
Pedro, pedro@example.com, $20.00
#...

Reading the csv

If you are sending the csv, with send_file or send_data in the controller, you can access the csv with page.body.

And you can get the CSV as an array, by parsing the page.body like this…

CSV.parse(page.body)

How to represent the csv data?

One way to represent the data in a readable way for your tests is to “transpose” the csv data to show the columns as rows, like this…

[
  ["User", "Juan", "Pedro"],
  ["Email", "juan@example.com", "pedro@example.com"],
  ["Amount", "$10.00", "$20.00"]
]

And to do this, you can call the transpose method…

CSV.parse(page.body).transpose

A test example

Putting it all together, a test could look like this…

RSpec.feature "My feature" do
  scenario do
    visit page_with_csv_button_path

    click_on "Download csv"

    expect(csv).to eq([
      ["User", "Juan", "Pedro"],
      ["Email", "juan@example.com", "pedro@example.com"],
      ["Amount", "$10.00", "$20.00"]
    ])
  end

  def csv
    CSV.parse(page.body).transpose
  end
end

And that’s all!… I hope it helps =)

A "read-more" behavior truncating with rails truncate helper and Stimulus.js

2021-04-19T00:28:00Z

A code sample of a “read-more” behavior using the rails truncate method and Stimulus.js.

The example

Visit example page

The html and css

Your are going to need…

A fullContentValue with the full content.
A class to hide the buttons.
3 targets, the content, the moreButton and the lessButton…
2 actions
- click->read-more#showMore
- click->read-more#showLess

.hide { display: none; }

<div data-controller="read-more"
     data-read-more-full-content-value="This is the full content without truncation"
     data-read-more-hide-class="hide">
  <p data-read-more-target="content">
    This is the truncated....
  </p>
  <button class="hide"
    data-read-more-target="moreButton"
    data-action="read-more#showMore">
    Show more
  </button>
  <button class="hide"
    data-read-more-target="lessButton"
    data-action="read-more#showLess">
    Show less
  </button>
</div>

To truncate the content, you can use the truncate helper.

truncate("Once upon a time in a world far far away")

The javascript

Here you are going to toggle the truncated content and the full content as needed.

class ReadMoreController extends Stimulus.Controller {
  static get targets() {
    return ["content", "moreButton", "lessButton"]
  }

  static get classes() {
    return ["hide"]
  }

  static get values() {
    return { fullContent: String }
  }

  connect() {
    this.truncatedContent = this.contentTarget.textContent.trim();
    this.render()
  }

  render() {
    if (this.truncateContent != this.fullContentValue) {
      this.showLess()
    } else {
      this.hide(this.moreButtonTarget);
      this.hide(this.lessButtonTarget);
    }
  }

  showMore() {
    this.contentTarget.textContent = this.fullContentValue;
    this.hide(this.moreButtonTarget);
    this.show(this.lessButtonTarget);
  }

  showLess() {
    this.contentTarget.textContent = this.truncateContent;
    this.hide(this.lessButtonTarget);
    this.show(this.moreButtonTarget);
  }

  show(target) {
    target.classList.remove(this.hideClass)
  }

  hide(target) {
    target.classList.add(this.hideClass)
  }
}

Other ways of doing it…

If instead of number characters or words you need to truncate your content by number of lines, you can try these examples.

A "read-more" behavior truncating by number of lines with custom javascript and Stimulus.js

2021-04-02T00:28:00Z

Are you looking for a way of implementing a “read-more” behavior truncating by the number of lines instead of the number of words?

If this is your case, this article can help you =) …

The example

Visit example page

The html and css

Your are going to need…

A linesValue to configure the number of lines that we want to display,
A class to hide the buttons…
3 targets, the content, the moreButton and the lessButton…
3 actions
- resize@window->read-more#render - to calculate the truncation on rezise.
- click->read-more#showMore
- click->read-more#showLess

html { line-height: 1.5 }
.hide { display: none; }

<div data-controller="read-more"
     data-read-more-lines-value="3"
     data-read-more-hide-class="hide"
     data-action="resize@window->read-more#render">
  <p data-read-more-target="content">
    The content that you want to truncate...
  </p>
  <button class="hide"
    data-read-more-target="moreButton"
    data-action="read-more#showMore">
    Show more
  </button>
  <button class="hide"
    data-read-more-target="lessButton"
    data-action="read-more#showLess">
    Show less
  </button>
</div>

The javascript

On render you are going to truncate the content, to the configured linesValue.

But you are going to do it just if the height is greater that the expectedHeigt that is the product of the linesValue with the lineHeight.

class ReadMoreController extends Stimulus.Controller {
  static get targets() {
    return ["content", "moreButton", "lessButton"]
  }

  static get classes() {
    return ["hide"]
  }

  static get values() {
    return { lines: Number }
  }

  connect() {
    this.content = this.contentTarget.textContent;
    this.render()
  }

  render() {
    this.showAllContent()

    if (this.height() > this.expectedHeight()) {
      this.showLess()
    } else {
      this.showAllContent()
      this.hide(this.moreButtonTarget);
      this.hide(this.lessButtonTarget);
    }
  }

  showMore() {
    this.showAllContent();
    this.hide(this.moreButtonTarget);
    this.show(this.lessButtonTarget);
  }

  showLess() {
    this.truncateContent();
    this.hide(this.lessButtonTarget);
    this.show(this.moreButtonTarget);
  }

  showAllContent() {
    this.removeContent();
    this.wordsList().forEach((word) => this.addWordToContent(word))
  }

  truncateContent() {
    this.calculateWordsToDisplayWhenTruncated();
    this.renderTrucatedContentWithEllipsis();
  }

  calculateWordsToDisplayWhenTruncated() {
    this.wordsToDisplayWhenTrucated = [];
    this.removeContent();
    this.wordsList().forEach((word) => {
      if (this.height() < this.expectedHeight()) {
        this.wordsToDisplayWhenTrucated.push(word)
        this.addWordToContent(word)
      }
    })
  }

  renderTrucatedContentWithEllipsis() {
    this.wordsToDisplayWhenTrucated.pop()
    this.removeContent();
    this.wordsToDisplayWhenTrucated.forEach((word) => this.addWordToContent(word))
    this.addToContent("...")

    if (this.height() > this.expectedHeight()) {
      this.renderTrucatedContentWithEllipsis()
    }
  }

  show(target) {
    target.classList.remove(this.hideClass)
  }

  hide(target) {
    target.classList.add(this.hideClass)
  }

  removeContent() {
    this.contentTarget.textContent = "";
  }

  addWordToContent(word) {
    this.addToContent(" " + word);
  }

  addToContent(text) {
    this.contentTarget.textContent += text
  }

  lineHeight() {
    let style = window.getComputedStyle(this.contentTarget)
    return parseFloat(style.lineHeight, 10);
  }

  height() {
    return this.contentTarget.offsetHeight;
  }

  expectedHeight() {
    return this.linesValue * this.lineHeight();
  }

  wordsList() {
    return this.content.split(" ")
  }
}

Other ways of doing it…

If you are looking for a way to do the truncation using css line-clamp, you take a look to this example using “line-clamp”.

And if you can truncate by number of characters, you can check this example using the “truncate” helper.

Capybara cheatsheet

2021-03-24T01:53:00Z

Are you working with capybara?… Are you constantly searching for capybara helpers?

Here is a small reference with some of the most used methods/helpers.

# Navigate to a particular path
visit blogs_path
visit "/blog"

# Click on an anchor tag, button, or input with type submit
click_on "Save Project"
click_on "New Project"

# Click an anchor tag
click_link 'New Project'

# Click a button, or input with type submit
click_button 'Save'

# Find and click a Capybara::Element
find("a.title").click

Form interaction

# Fill input fields (text, textarea, number, phone, etc...)
fill_in 'Title', with: 'I love Rails!'
fill_in 'post[title]', with: 'I love Rails!'

# Checkbox
check 'I accept the terms and conditions'
uncheck 'I accept the terms and conditions'

# Radio button
choose 'Female'

# Select option from select tag
select 'MA', from: 'State'

# File input
attach_file Rails.root.join('spec/fixture/some_file.png')

# Click a button, or input with type submit
click_button 'Save'

# Accept a confirm dialog
accept_confirm do
  click_on 'Destroy'
end

Page interaction and scoping

# Returns a single Capybara::Node::Element instance from the page.
# It will wait for an element to appear on the page.
# If the element does not appear it will raise an error.
page.find('.todos li:first-child')

# Returns an array of Capybara::Node::Element instances from the page.
page.all('.todos li:nth-of-type(odd)')

# Within will scope interaction to within a particular selector.
within('footer') do
  expect(page).to have_content('Copyright')
end

# Is the selector present on the page?
page.has_css?('nav[data-role="user-menu"] li', text: 'Profile')

# Is the content present on the page?
page.has_content?('Sign in')

Matchers

# Is the content present on the page?
expect(page).to have_content("Some Content")
expect(page).to have_no_content("Some Content")

# Is the selector present on the page?
expect(page).to have_css("input#post_title")
expect(page).to have_css("input#post_title", value: "Capybara cheatsheet")

# Is the selector present X times on the page?
expect(page).to have_css("input", count: 3)
expect(page).to have_css("input", maximum: 3)
expect(page).to have_css("input", minimum: 3)
expect(page).to have_css("input", between: 1..3)

# Is the selector with the right text on the page?
expect(page).to have_css("p a", text: "hello")
expect(page).to have_css("p a", text: /[hH]ello(.+)/i)

# Is the field present?
expect(page).to have_field("FirstName")
expect(page).to have_field("FirstName", value: "Rambo")
expect(page).to have_field("FirstName", with: "Rambo")
expect(page).to have_field("FirstName", disabled: true)

# Is the link present?
expect(page).to have_link("Foo")
expect(page).to have_link("Foo", href: "example.com")
expect(page).to have_no_link("Foo", href: "example.com")

Node interaction

# Trigger a click on a Capybara::Element
find("a.title").click

# Trigger allow triggering a custom event.
find("a.title").trigger("focus")

# Is the element visible?
find(".navigation").visible?

Debugging

# Save the current page and attempt to open the HTML
# in the default web browser.
save_and_open_page

References

This cheatsheet is just a mix of other very usefull cheatsheets and other references.

Get the cheatsheet on PDF

If you work with capybara and you are constantly searching for capybara helpers, maybe to have this little cheatsheet at hand could work for you.

Custom capybara matchers with compound expectations

2021-03-17T13:12:00Z

Imagine that you want to refactor this…

expect(row).to have_css("[data-role=description]", text: "Desc 1")
expect(row).to have_css("[data-role=amount]", text: "$250.00")

into this…

expect(row).to display_entry_with("Desc1", $250.00)

What would you do?…

Maybe you can write a custom matcher, but at least for me, in cases like this, it just feel like too much.

One thing that I think is simpler is to use rspec’s compound expectations, to chain all the expectations that you want, and return a matcher.

Something like this…

def display_entry_with(description, amount)
  have_css("[data-role=description]", text: description)
    .and have_css("[data-role=amount]", text: amount)
end

I have found this little trick very usefull, I hope it can help you too.

Quiz to practice the difference between preload, includes or eager_load

2021-02-22T13:05:00Z

Do you need more practice to understand the difference between “includes”, “preload”, “eager_load” and “joins”?

Here you will find a set of exercises to help you understand how you can use this 4 methods.

You can find the code on bhserna/preloading_quiz

This is are the exercises for the post Examples to learn the difference between preload, includes or eager_load, you can find the answers there.

On /exercises you will find a list of exercises with:

The description of a task
And a function to write the results

You should provide a way to fetch the records.

For example:

puts "-----------------------------"
puts "TASK"
puts "1. Fetch all acomodations preloading rooms (with includes)"
puts "-----------------------------"

# accomodations = ?
# display_with_rooms(accomodations)

How to run the examples

Create a postgres database with createdb preloading_quiz. As you can see the in db/config.rb the name of the database is hardcoded, so you will need to create a database with that name.
Install the dependencies with bundle install.
Run the seeds with ruby db/seeds.rb. You can also update the data that you want to use, there.
Run the exercises with ruby exercises/01.rb, etc…

Which specs would you prefer for this presentation object?

2021-02-17T16:36:00Z

I am currently working on a kind of “presentation object” that I called IdealPayments::GlobalStats.

This object will be used in two places, to display a graph and to show the data in a page where we can see all the “ideal payments”.

In my imagination and implementation, this object will depenend on an active record relation and chain the right methods to present the result.

Although I think this is not a big deal, I would like to know what do you think is best way to write unit/micro tests for this object and why?

I have two options…

Test through the database as if the source (the active record relation) was invisible.
Test by allowing that “the right messages” will be send to the “source” and expect the right result is returned.

Note: I already have test all the methods on the active record class through the database.

Option 1 - Test through the database as if the source (the active record relation) was invisible.

module IdealPayments
  RSpec.describe GlobalStats do
    describe "#generated_amount" do
      it "is the sum of the ideal payments amount" do
        create :ideal_payment, amount: 1_000
        create :ideal_payment, amount: 2_000
        expect(subject.generated_amount).to eq 3_000
      end

      it "is just for projects promoted by us" do
        dev_project = create :dev_project, promoter: "other"
        project = create :project, dev_project: dev_project
        ideal_payments_for_project = create :ideal_payments_for_project, project: project

        create :ideal_payment, ideal_payments_for_project: ideal_payments_for_project, amount: 1_000
        create :ideal_payment, amount: 2_000
        expect(subject.generated_amount).to eq 2_000
      end
    end

    describe "#paid_amount" do
      it "is the sum of the ideal payments paid amount" do
        create :ideal_payment, paid_amount: 1_000
        create :ideal_payment, paid_amount: 1_000
        expect(subject.paid_amount).to eq 2_000
      end
    end

    describe "#pending_amount" do
      it "is the difference between the generated amount and the paid amount" do
        create :ideal_payment, amount: 1_000, paid_amount: 500
        create :ideal_payment, amount: 1_000, paid_amount: 200
        expect(subject.pending_amount).to eq 2_000 - 700
      end
    end

    describe "#pending_amount_without_delay" do
      it "is the pending amount that is not yet delayed" do
        create :ideal_payment, paid_amount: 0, amount: 1500, date: Date.current + 10.days
        create :ideal_payment, paid_amount: 0, amount: 1000, date: Date.current - 10.days
        create :ideal_payment, paid_amount: 1500, amount: 2000, date: Date.current
        expect(subject.pending_amount_without_delay).to eq 2000
      end
    end

    describe "#delayed_amount" do
      it "is the pending amount that is delayed" do
        create :ideal_payment, paid_amount: 0, amount: 1500, date: Date.current
        create :ideal_payment, paid_amount: 0, amount: 1000, date: Date.current - 10.days
        create :ideal_payment, paid_amount: 1500, amount: 2000, date: Date.current - 5.days
        expect(subject.delayed_amount).to eq 1500
      end
    end

    describe "#delayed_amount_less_than_30_days" do
      it "is the pending amount that is delayed less than 30 days" do
        create :ideal_payment, paid_amount: 0, amount: 8000, date: Date.current + 10.days
        create :ideal_payment, paid_amount: 0, amount: 1000, date: Date.current - 30.days
        create :ideal_payment, paid_amount: 1500, amount: 2000, date: Date.current - 5.days
        expect(subject.delayed_amount_less_than_30_days).to eq 1500
      end
    end

    describe "#delayed_amount_between_30_and_90_days" do
      it "is the pending amount that is delayed more than 30 days but less than 90 days" do
        create :ideal_payment, paid_amount: 0, amount: 1500, date: Date.current - 90.days
        create :ideal_payment, paid_amount: 0, amount: 2500, date: Date.current - 30.days
        create :ideal_payment, paid_amount: 0, amount: 1000, date: Date.current - 50.days
        create :ideal_payment, paid_amount: 1500, amount: 2000, date: Date.current - 5.days
        expect(subject.delayed_amount_between_30_and_90_days).to eq 2500
      end
    end

    describe "#delayed_amount_more_90_days" do
      it "is the pending amount that is delayed more than than 90 days" do
        create :ideal_payment, paid_amount: 0, amount: 1500, date: Date.current - 100.days
        create :ideal_payment, paid_amount: 0, amount: 1000, date: Date.current - 90.days
        create :ideal_payment, paid_amount: 1500, amount: 2000, date: Date.current - 95.days
        expect(subject.delayed_amount_more_90_days).to eq 2000
      end
    end
  end
end

Option 2. Test by allowing that “the right messages” will be send to the “source” and expect the right result is returned.

module IdealPayments
  RSpec.describe GlobalStats do
    describe "#source" do
      it "is by default the ideal payments for projects promoted by us" do
        expected = double("Ideal payments for project promoted by us")
        allow(IdealPayment).to receive(:for_projects_promoted_by_us).and_return(expected)
        expect(subject.source).to eq expected
      end
    end

    describe "#generated_amount" do
      it "is the total amount" do
        allow(subject.source).to receive(:total_amount).and_return(3_000)
        expect(subject.generated_amount).to eq 3_000
      end
    end

    describe "#paid_amount" do
      it "is the total paid amount" do
        allow(subject.source).to receive(:total_paid_amount).and_return(2_000)
        expect(subject.paid_amount).to eq 2_000
      end
    end

    describe "#pending_amount" do
      it "is the total paid amount" do
        allow(subject.source).to receive(:total_pending_amount).and_return(1_000)
        expect(subject.pending_amount).to eq 1_000
      end
    end

    describe "#pending_amount_without_delay" do
      it "is the total pending amount of the not delayed payments" do
        allow(subject.source).to receive_message_chain(:not_delayed, :total_pending_amount).and_return(1_000)
        expect(subject.pending_amount_without_delay).to eq 1_000
      end
    end

    describe "#delayed_amount" do
      it "is the total pending amount of the delayed payments" do
        allow(subject.source).to receive_message_chain(:delayed, :total_pending_amount).and_return(1_000)
        expect(subject.delayed_amount).to eq 1_000
      end
    end

    describe "#delayed_amount_less_than_30_days" do
      it "is the total pending amount that is delayed less than 30 days" do
        delayed = double("Delayed")
        allow(subject.source).to receive(:delayed_less_than).with(30.days).and_return(delayed)
        allow(delayed).to receive(:total_pending_amount).and_return(1500)
        expect(subject.delayed_amount_less_than_30_days).to eq 1500
      end
    end

    describe "#delayed_amount_between_30_and_90_days" do
      it "is the total pending amount that is delayed more than 30 days but less than 90 days" do
        delayed = double("Delayed")
        allow(subject.source).to receive(:delayed_between).with(30.days, 90.days).and_return(delayed)
        allow(delayed).to receive(:total_pending_amount).and_return(2500)
        expect(subject.delayed_amount_between_30_and_90_days).to eq 2500
      end
    end

    describe "#delayed_amount_more_90_days" do
      it "is the pending amount that is delayed more than than 90 days" do
        delayed = double("Delayed")
        allow(subject.source).to receive(:delayed_more_than).with(90.days).and_return(delayed)
        allow(delayed).to receive(:total_pending_amount).and_return(3500)
        expect(subject.delayed_amount_more_90_days).to eq 3500
      end
    end
  end
end

Here is the actual implementation

module IdealPayments
  class GlobalStats
    def initialize(ideal_payments = IdealPayment.for_projects_promoted_by_us)
      @ideal_payments = ideal_payments
    end

    def source
      ideal_payments
    end

    def generated_amount
      ideal_payments.total_amount
    end

    def paid_amount
      ideal_payments.total_paid_amount
    end

    def pending_amount
      ideal_payments.total_pending_amount
    end

    def pending_amount_without_delay
      ideal_payments.not_delayed.total_pending_amount
    end

    def delayed_amount
      ideal_payments.delayed.total_pending_amount
    end

    def delayed_amount_less_than_30_days
      ideal_payments.delayed_less_than(30.days).total_pending_amount
    end

    def delayed_amount_between_30_and_90_days
      ideal_payments.delayed_between(30.days, 90.days).total_pending_amount
    end

    def delayed_amount_more_90_days
      ideal_payments.delayed_more_than(90.days).total_pending_amount
    end

    private

    attr_reader :ideal_payments
  end
end

Code to run the examples for preload, includes or eager_load

2021-02-16T00:18:00Z

Do you know “in theory” the difference between “includes”, “preload”, “eager_load” and “joins”, but you still think that you need more examples to really understand how to use them?

Here you will find a set of examples to help you understand how you can use this 4 methods.

You can find the code on bhserna/preloading_examples

This is the code that I used for the post: Examples to learn the difference between preload, includes or eager_load.

On lib/examples.rb you will find a list of examples with:

The description of a task
The query written with ActiveRecord
A function to write the results

You can use this code to play a little more and try different things.

How to run the examples

Create a postgres database with createdb preloading_examples. As you can see the in db/config.rb the name of the database is hardcoded, so you will need to create a database with that name.
Install the dependencies with bundle install.
Run the seeds with ruby db/seeds.rb. You can also update the data that you want to use, there.
Run the examples with ruby lib/examples.rb.

Examples to learn the difference between preload, includes or eager_load

2021-02-04T00:18:00Z

Here you will find a set of examples to help you understand how you can use this 4 methods.

For each example you will find:

The description of a task
The query written with ActiveRecord
The produced SQL query
The result with the fetched records

For each example/excercise try read the description the task and try to think a little before reading the answers.

The model of the example

All the examples are based on the next model…

ActiveRecord::Schema.define(version: 1) do
  create_table :accomodations, if_not_exists: true do |t|
    t.column :name, :string
    t.column :bathrooms_count, :decimal
    t.column :guests_count, :integer
  end

  create_table :rooms, if_not_exists: true do |t|
    t.column :accomodation_id, :integer
    t.column :name, :string
    t.column :beds_count, :integer
  end

  create_table :reviews, if_not_exists: true do |t|
    t.column :accomodation_id, :integer
    t.column :body, :text
    t.column :stars, :integer
  end
end

class Accomodation < ActiveRecord::Base
  has_many :rooms
  has_many :reviews
end

class Room < ActiveRecord::Base
end

class Review < ActiveRecord::Base
end

Examples

1. Fetch all acomodations preloading rooms (with includes)

ActiveRecord query

Accomodation.includes(:rooms)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "rooms".* FROM "rooms" WHERE "rooms"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Rooms
#<Room:0x00007f7fae488a20 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fae398408 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathroms_count: nil, guests_count: 4>
Rooms
#<Room:0x00007f7fae392fd0 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fae392968 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Rooms
#<Room:0x00007f7fae3926c0 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fae392238 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Rooms
#<Room:0x00007f7fae3920f8 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fae391b08 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

2. Fetch all acomodations preloading rooms (with preload)

ActiveRecord query

Accomodation.preload(:rooms)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "rooms".* FROM "rooms" WHERE "rooms"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Rooms
#<Room:0x00007f7fb28541a0 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fb284fec0 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Rooms
#<Room:0x00007f7fb284fda8 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fb284fbc8 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Rooms
#<Room:0x00007f7fb284fab0 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fb284f9c0 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Rooms
#<Room:0x00007f7fb284f790 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fb284f538 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

3. Fetch all acomodations preloading rooms (with eager load)

ActiveRecord query

Accomodation.eager_load(:rooms)

SQL query

SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "rooms"."id" AS t1_r0, "rooms"."accomodation_id" AS t1_r1, "rooms"."name" AS t1_r2, "rooms"."beds_count" AS t1_r3 FROM "accomodations" LEFT OUTER JOIN "rooms" ON "rooms"."accomodation_id" = "accomodations"."id"

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Rooms
#<Room:0x00007f7fae7b0e00 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fae7b0b80 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Rooms
#<Room:0x00007f7fae7b0108 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fae7abd88 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Rooms
#<Room:0x00007f7fae7ab5e0 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fae7ab270 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Rooms
#<Room:0x00007f7fae7aaaf0 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fae7aa7f8 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

4. Fetch all acomodations preloading reviews (with includes)

ActiveRecord query

Accomodation.includes(:reviews)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae75b018 id: 1, accomodation_id: 1, body: "B1", stars: 3>
#<Review:0x00007f7fae721160 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae721070 id: 3, accomodation_id: 1, body: "B3", stars: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae720cd8 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae720be8 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae720af8 id: 6, accomodation_id: 2, body: "B3", stars: 4>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae720a08 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fae720918 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae7207d8 id: 9, accomodation_id: 3, body: "B3", stars: 5>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae7206e8 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae7205f8 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae720508 id: 12, accomodation_id: 4, body: "B3", stars: 4>

5. Fetch all acomodations preloading reviews (with preload)

ActiveRecord query

Accomodation.preload(:reviews)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae6e8e00 id: 1, accomodation_id: 1, body: "B1", stars: 3>
#<Review:0x00007f7fae6e8d10 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae6e8ba8 id: 3, accomodation_id: 1, body: "B3", stars: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae6e8a18 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae6e88b0 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae6e86a8 id: 6, accomodation_id: 2, body: "B3", stars: 4>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae6e84c8 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fae6e82e8 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae6e80b8 id: 9, accomodation_id: 3, body: "B3", stars: 5>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae6e3f90 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae6e3ea0 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae6e3d88 id: 12, accomodation_id: 4, body: "B3", stars: 4>

6. Fetch all acomodations preloading reviews (with eager_load)

ActiveRecord query

Accomodation.eager_load(:reviews)

SQL query

SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "reviews"."id" AS t1_r0, "reviews"."accomodation_id" AS t1_r1, "reviews"."body" AS t1_r2, "reviews"."stars" AS t1_r3 FROM "accomodations" LEFT OUTER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id"

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae6a9e80 id: 1, accomodation_id: 1, body: "B1", stars: 3>
#<Review:0x00007f7fae6a9cf0 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae6a9a98 id: 3, accomodation_id: 1, body: "B3", stars: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae6a9638 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae6a9408 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae6a9250 id: 6, accomodation_id: 2, body: "B3", stars: 4>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae6a8d00 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fae6a8b98 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae6a8940 id: 9, accomodation_id: 3, body: "B3", stars: 5>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae6a8580 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae6a8328 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae6a8170 id: 12, accomodation_id: 4, body: "B3", stars: 4>

7. Fetch all acomodations preloading reviews and rooms (with includes)

ActiveRecord query

Accomodation.includes(:reviews, :rooms)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]
SELECT "rooms".* FROM "rooms" WHERE "rooms"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae66acd0 id: 1, accomodation_id: 1, body: "B1", stars: 3>
#<Review:0x00007f7fae66ab40 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae66aa50 id: 3, accomodation_id: 1, body: "B3", stars: 3>
Rooms
#<Room:0x00007f7fae662d78 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fae662c88 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae66a938 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae66a848 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae66a6b8 id: 6, accomodation_id: 2, body: "B3", stars: 4>
Rooms
#<Room:0x00007f7fae662b70 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fae662a80 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae66a438 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fae66a348 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae66a258 id: 9, accomodation_id: 3, body: "B3", stars: 5>
Rooms
#<Room:0x00007f7fae662990 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fae662788 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae66a168 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae66a078 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae669f88 id: 12, accomodation_id: 4, body: "B3", stars: 4>
Rooms
#<Room:0x00007f7fae662648 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fae662440 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

8. Fetch all acomodations preloading reviews and rooms (with preload)

ActiveRecord query

Accomodation.preload(:reviews, :rooms)

SQL query

SELECT "accomodations".* FROM "accomodations"
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]
SELECT "rooms".* FROM "rooms" WHERE "rooms"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 1], [nil, 2], [nil, 3], [nil, 4]]

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae6035f8 id: 1, accomodation_id: 1, body: "B1", stars: 3>
#<Review:0x00007f7fae603508 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae603418 id: 3, accomodation_id: 1, body: "B3", stars: 3>
Rooms
#<Room:0x00007f7fae5f9c60 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fae5f9af8 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae603328 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae6031c0 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae6030d0 id: 6, accomodation_id: 2, body: "B3", stars: 4>
Rooms
#<Room:0x00007f7fae5f99b8 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fae5f98a0 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae602fe0 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fae602ef0 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae602e00 id: 9, accomodation_id: 3, body: "B3", stars: 5>
Rooms
#<Room:0x00007f7fae5f97b0 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fae5f9698 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae602d10 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae602b58 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae602950 id: 12, accomodation_id: 4, body: "B3", stars: 4>
Rooms
#<Room:0x00007f7fae5f9580 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fae5f9468 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

9. Fetch all acomodations preloading reviews and rooms (with eager_load)

ActiveRecord query

Accomodation.eager_load(:reviews, :rooms)

SQL query

SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "reviews"."id" AS t1_r0, "reviews"."accomodation_id" AS t1_r1, "reviews"."body" AS t1_r2, "reviews"."stars" AS t1_r3, "rooms"."id" AS t2_r0, "rooms"."accomodation_id" AS t2_r1, "rooms"."name" AS t2_r2, "rooms"."beds_count" AS t2_r3 FROM "accomodations" LEFT OUTER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" LEFT OUTER JOIN "rooms" ON "rooms"."accomodation_id" = "accomodations"."id"

Results

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Reviews
#<Review:0x00007f7fae52b5e0 id: 3, accomodation_id: 1, body: "B3", stars: 3>
#<Review:0x00007f7fae52b108 id: 2, accomodation_id: 1, body: "B2", stars: 3>
#<Review:0x00007f7fae52ae60 id: 1, accomodation_id: 1, body: "B1", stars: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae52a488 id: 6, accomodation_id: 2, body: "B3", stars: 4>
#<Review:0x00007f7fae529d58 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae529ad8 id: 4, accomodation_id: 2, body: "B1", stars: 4>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fae529218 id: 9, accomodation_id: 3, body: "B3", stars: 5>
#<Review:0x00007f7fae528b38 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fae5289a8 id: 7, accomodation_id: 3, body: "B1", stars: 5>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae5280e8 id: 12, accomodation_id: 4, body: "B3", stars: 4>
#<Review:0x00007f7fae523a20 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae5235e8 id: 10, accomodation_id: 4, body: "B1", stars: 5>

10. Fetch all acomodations for exactly 4 guests, preloading reviews (with includes)

ActiveRecord query

Accomodation.where(guests_count: 4).includes(:reviews)

SQL query

SELECT "accomodations".* FROM "accomodations" WHERE "accomodations"."guests_count" = $1  [["guests_count", 4]]
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" = $1  [["accomodation_id", 2]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae4a3640 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae4a2f38 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae4a2830 id: 6, accomodation_id: 2, body: "B3", stars: 4>

11. Fetch all acomodations for exactly 4 guests, preloading reviews (with preload)

ActiveRecord query

Accomodation.where(guests_count: 4).preload(:reviews)

SQL query

SELECT "accomodations".* FROM "accomodations" WHERE "accomodations"."guests_count" = $1  [["guests_count", 4]]
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" = $1  [["accomodation_id", 2]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae439678 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae4391c8 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae438f70 id: 6, accomodation_id: 2, body: "B3", stars: 4>

12. Fetch all acomodations for exactly 4 guests, preloading reviews (with eager_load)

ActiveRecord query

Accomodation.where(guests_count: 4).eager_load(:reviews)

SQL query

SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "reviews"."id" AS t1_r0, "reviews"."accomodation_id" AS t1_r1, "reviews"."body" AS t1_r2, "reviews"."stars" AS t1_r3 FROM "accomodations" LEFT OUTER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" WHERE "accomodations"."guests_count" = $1  [["guests_count", 4]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae3d2d88 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae3d25b8 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae3d1dc0 id: 6, accomodation_id: 2, body: "B3", stars: 4>

13. Fetch all acomodations with reviews of 4 stars (not in average), preloading reviews (with includes)

ActiveRecord query

Accomodation.includes(:reviews).where(reviews: {stars: 4})

SQL query

SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "reviews"."id" AS t1_r0, "reviews"."accomodation_id" AS t1_r1, "reviews"."body" AS t1_r2, "reviews"."stars" AS t1_r3 FROM "accomodations" LEFT OUTER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" WHERE "reviews"."stars" = $1  [["stars", 4]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae359a50 id: 6, accomodation_id: 2, body: "B3", stars: 4>
#<Review:0x00007f7fae359708 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae359280 id: 4, accomodation_id: 2, body: "B1", stars: 4>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae353808 id: 12, accomodation_id: 4, body: "B3", stars: 4>
#<Review:0x00007f7fae3534c0 id: 11, accomodation_id: 4, body: "B2", stars: 4>

14. Fetch all acomodations with reviews of 4 stars (not in average), preloading reviews (with preload)

ActiveRecord query

Accomodation.joins(:reviews).where(reviews: {stars: 4}).distinct(true).preload(:reviews)

SQL query

SELECT DISTINCT "accomodations".* FROM "accomodations" INNER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" WHERE "reviews"."stars" = $1  [["stars", 4]]
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2)  [[nil, 2], [nil, 4]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fae27a828 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fae279928 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fae2737f8 id: 6, accomodation_id: 2, body: "B3", stars: 4>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fae2721f0 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fae26a400 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fae269848 id: 12, accomodation_id: 4, body: "B3", stars: 4>

15. Fetch all acomodations with reviews of 4 stars (not in average), preloading reviews (with eager_load)

ActiveRecord query

Accomodation.eager_load(:reviews).where(reviews: {stars: 4})

SQL query

SQL (0.3ms)  SELECT "accomodations"."id" AS t0_r0, "accomodations"."name" AS t0_r1, "accomodations"."bathrooms_count" AS t0_r2, "accomodations"."guests_count" AS t0_r3, "reviews"."id" AS t1_r0, "reviews"."accomodation_id" AS t1_r1, "reviews"."body" AS t1_r2, "reviews"."stars" AS t1_r3 FROM "accomodations" LEFT OUTER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" WHERE "reviews"."stars" = $1  [["stars", 4]]

Results

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fb28acee0 id: 6, accomodation_id: 2, body: "B3", stars: 4>
#<Review:0x00007f7fb28accd8 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fb28acb48 id: 4, accomodation_id: 2, body: "B1", stars: 4>

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fb28ac760 id: 12, accomodation_id: 4, body: "B3", stars: 4>
#<Review:0x00007f7fb28ac580 id: 11, accomodation_id: 4, body: "B2", stars: 4>

16. List of accomodations with reviews of 4 stars (in average), preloading reviews

ActiveRecord query

Accomodation.joins(:reviews).group(:id).having("avg(reviews.stars) > 4").preload(:reviews)

SQL query

SELECT "accomodations".* FROM "accomodations" INNER JOIN "reviews" ON "reviews"."accomodation_id" = "accomodations"."id" GROUP BY "accomodations"."id" HAVING (avg(reviews.stars) >= 4)
SELECT "reviews".* FROM "reviews" WHERE "reviews"."accomodation_id" IN ($1, $2, $3)  [[nil, 4], [nil, 2], [nil, 3]]

Results

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Reviews
#<Review:0x00007f7fb28747e8 id: 10, accomodation_id: 4, body: "B1", stars: 5>
#<Review:0x00007f7fb28746f8 id: 11, accomodation_id: 4, body: "B2", stars: 4>
#<Review:0x00007f7fb2874608 id: 12, accomodation_id: 4, body: "B3", stars: 4>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Reviews
#<Review:0x00007f7fb2875a80 id: 4, accomodation_id: 2, body: "B1", stars: 4>
#<Review:0x00007f7fb2874d38 id: 5, accomodation_id: 2, body: "B2", stars: 4>
#<Review:0x00007f7fb2874c48 id: 6, accomodation_id: 2, body: "B3", stars: 4>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Reviews
#<Review:0x00007f7fb2874b58 id: 7, accomodation_id: 3, body: "B1", stars: 5>
#<Review:0x00007f7fb28749c8 id: 8, accomodation_id: 3, body: "B2", stars: 5>
#<Review:0x00007f7fb28748d8 id: 9, accomodation_id: 3, body: "B3", stars: 5>

17. List of accomodations with exactly 2 rooms, preloading rooms

ActiveRecord query

Accomodation.joins(:rooms).group(:id).having("count(rooms.id) = 2").preload(:rooms)

SQL query

SELECT "accomodations".* FROM "accomodations" INNER JOIN "rooms" ON "rooms"."accomodation_id" = "accomodations"."id" GROUP BY "accomodations"."id" HAVING (count(rooms.id) = 2)
SELECT "rooms".* FROM "rooms" WHERE "rooms"."accomodation_id" IN ($1, $2, $3, $4)  [[nil, 4], [nil, 2], [nil, 3], [nil, 1]]

Results

#<Accomodation id: 4, name: "Home4", bathrooms_count: nil, guests_count: 7>
Rooms
#<Room:0x00007f7fb282dd48 id: 7, accomodation_id: 4, name: "R1", beds_count: 6>
#<Room:0x00007f7fb282dc08 id: 8, accomodation_id: 4, name: "R2", beds_count: 3>

#<Accomodation id: 2, name: "Home2", bathrooms_count: nil, guests_count: 4>
Rooms
#<Room:0x00007f7fb282e540 id: 3, accomodation_id: 2, name: "R1", beds_count: 4>
#<Room:0x00007f7fb282e338 id: 4, accomodation_id: 2, name: "R2", beds_count: 3>

#<Accomodation id: 3, name: "Home3", bathrooms_count: nil, guests_count: 5>
Rooms
#<Room:0x00007f7fb282e1d0 id: 5, accomodation_id: 3, name: "R1", beds_count: 6>
#<Room:0x00007f7fb282df28 id: 6, accomodation_id: 3, name: "R2", beds_count: 3>

#<Accomodation id: 1, name: "Home1", bathrooms_count: nil, guests_count: 3>
Rooms
#<Room:0x00007f7fb282e950 id: 1, accomodation_id: 1, name: "R1", beds_count: 2>
#<Room:0x00007f7fb282e680 id: 2, accomodation_id: 1, name: "R2", beds_count: 3>

Did you finish?

If you have already finished… Awesome!!

If you need more time… It’s ok!… Try to make some time in your schedule and try to finish it!

If you find this post helpful, it would be nice if you can share it with someone that you think it also could be helpful =).

Do you want the code?

You can find this examples in two ways.

A repo with all the examples with the answers - You will be able to run the code, and experiment with all the exercises.
A repo just the exercises, as a quiz - You will be able to test you knowledge and experiment a little more.

How to attach an active storage file to an email

2021-01-27T13:00:00Z

Are you using Active Storage to handle file uploading to the cloud (maybe AWS) and you want to know how to attach an uploaded file to an email?… But you are not familiar with the Active Storage and Action Mailer APIs?

I answered this question on reddit some days ago =)… Here is how you can do it…

You need to download the file and assign as and attachment, something like this…

attachments[photo.filename.to_s] = photo.download

Here the context of the example…

class Product < ApplicationRecord
  has_one_attached :photo
end

class PhotoMailer < ApplicationMailer
  default from: "default@example.com"

  def new_photo_email
    photo = params[:product].photo
    attachments[photo.filename.to_s] = photo.download
    mail(to: "one@example.com", subject: "Nueva photo")
  end
end

class PhotoMailerPreview < ActionMailer::Preview
  def new_photo_email
    PhotoMailer.with(product: Product.first).new_photo_email
  end
end

Here the reference from the rails guides https://edgeguides.rubyonrails.org/activestorageoverview.html#downloading-files

Why using “download” works?

If you are not familiar with Action Mailer and you are reading the docs, maybe is not that clear what you need to do. Here is what I understand.

On the Action Mailer docs you can see…

attachments['filename.jpg'] = File.read('/path/to/filename.jpg')

Inside the [] you need to put the name that you want for the attachment and you need to assign the content of the file. Like this:

attachments[filename] = file_content

In the example from the docs, File.read returns the content of the file.

Now, in the ActiveStorage docs says that you can “Use the attachment’s download method to read a blob’s binary data into memory”, and they use this example:

binary = user.avatar.download

That’s why I propose you to use download. Apparently download will return the content of the file wherever the file is stored.

When is better to use preload or eager_load vs includes?

2021-01-19T00:09:00Z

Even if you already know what is the difference between includes, preload and eager_load, maybe is not that clear when is better for you to use one vs the other.

If you don’t know the difference you can read… What is the difference between includes, preload, eager_load and joins?

Well, as you may know, with Rails/ActiveRecord you can preload associations in 2 different ways:

With a LEFT OUTER JOIN to pull all the associations in one query (eager_load).
Making a separate query for each association (preload)

Using `includes` by default

When you use includes Rails will decide which strategy to use for you. I think that this is a good default. Use includes and let rails decide for you.

But, to be honest, that is not what I do…

Using `preload` by default

I (at least for now) prefer to use preload explicitly by default, because that is also the default for includes, and I think that is easier to know what is going to happen just by watching the code.

I use eager_load just when I already know that is significantly faster than the preload.

So, what should you do?

I would say, “Don’t worry to much”, and just pick one of this two strategies.

Do you want to practice?

Practice will help you understand this concepts better. I have prepared some examples/excercises to help you understand how you can use this methods. Give it try!

Examples to learn the difference between preload, includes or eager_load

What is the difference between includes, preload, eager_load and joins in ActiveRecord?

2021-01-07T22:22:00Z

Are you dealing with includes, preload, eager_load and joins, maybe fixing some n+1 queries, but you are not really sure what’s the difference between all of them?

Would you like to be able to say things like… “Here I will use preload instead of includes because I have tested it and in this case an INNER JOIN is faster than a LEFT OUTER JOIN”

Well, here is a little guide to understand what they do and what is the difference between them, to help you decide which is better for your case.

Example data

To explain the concepts imagine that you have the next data in your database

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

post = Post.create(title: "Post 1", body: "Post 1 body")
post.comments.create(body: "Comment 1")
post.comments.create(body: "Comment 2")

post = Post.create(title: "Post 2", body: "Post 2 body")
post.comments.create(body: "Comment 1")
post.comments.create(body: "Comment 2")

Preload

It preloads the associociations using different queries.

For example, this code will use one query for the posts and other for the comments.

posts = Post.preload(:comments).to_a
# SELECT "posts".* FROM "posts"
# SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2)  [[nil, 1], [nil, 2]]

And you will be able to use the comments without n+1 queries.

posts.map(&:comments)

You can include conditions for the base model.

posts = Post.preload(:comments).where(title: "Post 1").to_a
# SELECT "posts".* FROM "posts" WHERE "posts"."title" = $1  [["title", "Post 1"]]
# SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2)  [[nil, 1], [nil, 2]]

But you can’t add conditions for the preloaded associations.

Post.preload(:comments).where(comments: {body: "Hola"}).to_a
# `prepare': PG::UndefinedTable: ERROR:  missing FROM-clause entry for table "comments" (ActiveRecord::StatementInvalid)
# LINE 1: SELECT "posts".* FROM "posts" WHERE "comments"."body" = $1

Eager load

It forces the eager loading performing a LEFT OUTER JOIN.

For example, this code will use just one query to load both, the posts and the comments.

posts = Post.eager_load(:comments).to_a
# SELECT "posts"."id" AS t0_r0, "posts"."title" AS t0_r1, "posts"."body" AS t0_r2, "comments"."id" AS t1_r0, "comments"."body" AS t1_r1, "comments"."post_id" AS t1_r2 FROM "posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id"

And you will be able to use the comments without n+1 queries.

posts.map(&:comments)

You will be able to add conditions for the base model.

Post.eager_load(:comments).where(title: "Hola").to_a
# SELECT "posts"."id" AS t0_r0, "posts"."title" AS t0_r1, "posts"."body" AS t0_r2, "comments"."id" AS t1_r0, "comments"."body" AS t1_r1, "comments"."post_id" AS t1_r2 FROM "posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" WHERE "posts"."title" = $1  [["title", "Hola"]]

And you also will be able to add conditions for the associated models.

Post.eager_load(:comments).where(comments: {body: "Hola"}).to_a
# SELECT "posts"."id" AS t0_r0, "posts"."title" AS t0_r1, "posts"."body" AS t0_r2, "comments"."id" AS t1_r0, "comments"."body" AS t1_r1, "comments"."post_id" AS t1_r2 FROM "posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" WHERE "comments"."body" = $1  [["body", "Hola"]]

Includes

By default it will load the included associations using different queries. Like preload.

For example, this code will use one query for the posts and other for the comments.

posts = Post.includes(:comments).to_a
# SELECT "posts".* FROM "posts"
# SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2)  [[nil, 1], [nil, 2]]

And you will be able to use the comments without new queries.

posts.map(&:comments)

You will also be able to include add conditions for the base model.

# Post.includes(:comments).where(title: "Hola").to_a
# SELECT "posts".* FROM "posts" WHERE "posts"."title" = $1  [["title", "Post 1"]]
# SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2)  [[nil, 1], [nil, 2]]

And in this case, you will also be able to add conditions for the included associations. Because it will perform a LEFT OUTER JOIN like eager load does.

Post.includes(:comments).where(comments: {body: "Hola"}).to_a
# SELECT "posts"."id" AS t0_r0, "posts"."title" AS t0_r1, "posts"."body" AS t0_r2, "comments"."id" AS t1_r0, "comments"."body" AS t1_r1, "comments"."post_id" AS t1_r2 FROM "posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" WHERE "comments"."body" = $1  [["body", "Hola"]]

Note that if you want to pass the coditions as strings, you will need to use the references method, like this…

Post.includes(:comments).where("comments.body = ?", "Hola"}).references(:comments).to_a

So, includes is like a mix between preload and eager_load.

Joins

By default it will perform an INNER JOIN that you could use to create a more specific query using the joined tables.

For example you can ask for the comments for a post with an specific title, like this…

comments = Comment.joins(:post).where(post: {title: "Post 1"}).to_a
# SELECT "comments".* FROM "comments" INNER JOIN "posts" post ON post."id" = "comments"."post_id" WHERE "post"."title" = $1  [["title", "Post 1"]]

But it will not preload the post for each comment.

# comments.map(&:post)
# SELECT "posts".* FROM "posts" WHERE "posts"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
# SELECT "posts".* FROM "posts" WHERE "posts"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]

If you want to preload the post, you would need to use one of the methods above (includes, preload, eager load)…

For example…

comments = Comment.joins(:post).where(post: {title: "Post 1"}).preload(:post).to_a
# SELECT "comments".* FROM "comments" INNER JOIN "posts" post ON post."id" = "comments"."post_id" WHERE "post"."title" = $1  [["title", "Post 1"]]
# SELECT "posts".* FROM "posts" WHERE "posts"."id" = $1  [["id", 2]]

And now you will avoid the n+1 queries when you do…

comments.map(&:post)

Do you want more examples?

Maybe more examples will help you understand this concepts better. I have prepared some examples to help you understand how you can use this 4 methods. Give it look!

Examples to learn the difference between preload, includes or eager_load

Do you want to practice?

Practice will help you understand this concepts better. I have prepared some examples/excercises to help you understand how you can use this 4 methods. Give it try!

Quiz to practice the difference between preload, includes or eager_load

Useful links

You can use the rails docs to learn more:

5 ways to fetch the latest-N-of-each record on Rails

2020-12-23T22:22:00Z

A common cause of n+1 queries is fetching the “latest-N-of-each” record on a list of records.

Some examples of this problem are trying to get…

The latest comments for each post
The latest payments for each customer
The latest review for each product
The better prices for each product
Etc…

The problem is that once you have detected the n+1 queries, is not that easy to avoid them. Here I will show you 5 ways to solve this problem avoiding the n+1 queries.

But first, let’s explain the problem with an example…

Example problem

Fetch the latest 3 comments for each post

Imagine, that you have a model like this…

class Post < ActiveRecord::Base
  has_many :comments

  def latest_comments
    comments.order(id: :desc).limit(3)
  end
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

How do you render the latest comments for each post, without n+1 queries?…

Post.all.each do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

… you can’t just includes(:comments) and call latest_comments, because active record will “ignore your includes”.

Well, here are 5 ways to do it…

Note: If you want to fetch just the “latest-for-each”. Maybe you want to take a look to this post: 5 ways to fix the latest-comment n+1 problem

Do you want to run the examples?

You can get the code on: Tool to run the examples for the 5 ways to fetch the latest-n-of-each record

You can use it to run the examples and play with the code and seed values, to pick the right solution for you current case.

1. An object to fetch all and select just the latest N

You can create a new object to fetch all the comments, group them by post and then return just the latest 3 comments for each post.

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

class LatestCommentsForPosts
  def initialize(posts)
    @comments = scope_for(posts).group_by(&:post_id)
  end

  def for_post(post)
    (@comments[post.id] || []).first(3)
  end

  private

  def scope_for(posts)
    Comment.order(id: :desc).where(post_id: posts)
  end
end

posts = Post.limit(10)
comments = LatestCommentsForPosts.new(posts)

posts.each do |post|
  comments.for_post(post).each do |comment|
    puts comment.body
  end
end

2. An object to fetch just the latest N for each record

Using window functions

If fetching all the comments for each post is causing you memory or performance problems, you can use “Window functions” to ask the database for just the latest 3 comments for each post, group them by post and then return the comments for each post.

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

class LatestCommentsForPosts
  def initialize(posts)
    @comments = scope_for(posts).group_by(&:post_id)
  end

  def for_post(post)
    @comments[post.id] || []
  end

  private

  def scope_for(posts)
    ranked_comments = Comment.where(post_id: posts).select(<<-SQL)
      comments.*,
      dense_rank() OVER (
        PARTITION BY comments.post_id
        ORDER BY comments.id DESC
      ) AS comment_rank
    SQL

    Comment.from(ranked_comments, "comments").where("comment_rank <= 3")
  end
end

posts = Post.limit(10)
comments = LatestCommentsForPosts.new(posts)

posts.each do |post|
  comments.for_post(post).each do |comment|
    puts comment.body
  end
end

Using lateral join

Other approach that you could try is using a “Lateral join” to fetch just the latest 3 comments for each post, group them by post and then return the comments for each post.

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

class LatestCommentsForPosts
  def initialize(posts)
    @comments = scope_for(posts).group_by(&:post_id)
  end

  def for_post(post)
    @comments[post.id] || []
  end

  private

  def scope_for(posts)
    latest_comments = Post
      .select("latest_comments.*")
      .where(id: posts)
      .joins(<<-SQL)
        JOIN LATERAL (
          SELECT * FROM comments
          WHERE post_id = posts.id
          ORDER BY id DESC LIMIT 3
        ) AS latest_comments ON TRUE
      SQL

    Comment.from(latest_comments, "comments").order(id: :desc)
  end
end

posts = Post.limit(10)
comments = LatestCommentsForPosts.new(posts)

posts.each do |post|
  comments.for_post(post).each do |comment|
    puts comment.body
  end
end

3. An association to fetch just the latest N for each record

You can also represent the latest comments as an association…

Using window functions

You can use “"Window functions” to build a scope for just the latest 3 comments for each post, and pass that scope to the association’s definition.

class Post < ActiveRecord::Base
  has_many :comments
  has_many :latest_comments,
    -> { Comment.latest_comments_for_posts },
    class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post

  def self.latest_comments_for_posts
    ranked_comments = Comment.select(<<-SQL)
      comments.*,
      dense_rank() OVER (
        PARTITION BY comments.post_id
        ORDER BY comments.id DESC
      ) AS comment_rank
    SQL

    from(ranked_comments, "comments").where("comment_rank <= 3")
  end
end


posts = Post.limit(10).includes(:latest_comments)

posts.each do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

Using lateral join

You can use a “Lateral join” to build an scope for just the latest 3 comments for each post, and pass that scope to the association’s definition.

class Post < ActiveRecord::Base
  has_many :comments
  has_many :latest_comments,
    -> { Comment.latest_comments_for_posts },
    class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post

  def self.latest_comments_for_posts
    latest_comments = Post
      .select("latest_comments.*")
      .joins(<<-SQL)
        JOIN LATERAL (
          SELECT * FROM comments
          WHERE post_id = posts.id
          ORDER BY id DESC LIMIT 3
        ) AS latest_comments ON TRUE
      SQL

    Comment.from(latest_comments, "comments").order(id: :desc)
  end
end


posts = Post.limit(10).includes(:latest_comments)

posts.each do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

4. Russian doll caching

If you are already using fragment cache on your app, you can embrace the “Russian doll caching” and think of n+1 as a feature, let the n+1 queries run just the first time and use the cache after that.

class Post < ActiveRecord::Base
  has_many :comments

  def latest_comments
    comments.order(id: :desc).limit(3)
  end
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

In your view…

<% #posts/index.html.erb %>
<%= render partial: 'posts/post', collection: @posts, cached: true %>

<% #posts/_post.html.erb %>
<% cache post %>
  <%= post.title %>
  <% post.latest_comments.each do |comment| %>
    <%= comment.body %>
  <% end %>
<% end %>

5. A new record to cache the latest N for each record

If the other solutions are not enough for you, you can try to add a new record to keep a cache of the latest comments for each post using a has_many :through association and keep the latest comments in sync “manually” (maybe with an after_commit callback).

class Post < ActiveRecord::Base
  has_many :comments
  has_many :latest_comment_caches, class_name: "LatestCommentCache"
  has_many :latest_comments, -> { order(id: :desc) },
    through: :latest_comment_caches,
    source: :comment

  def delete_latest_comment_caches
    latest_comment_caches.delete_all
  end

  def calculate_latest_comments
    comments.order(id: :desc).limit(3)
  end
end

class Comment < ActiveRecord::Base
  belongs_to :post
  after_commit -> { LatestCommentCache.update_for_post(post) }
end

class LatestCommentCache < ActiveRecord::Base
  belongs_to :post
  belongs_to :comment

  def self.update_for_post(post)
    post.delete_latest_comment_caches

    post.calculate_latest_comments.each do |comment|
      create(post: post, comment: comment)
    end
  end
end

posts = Post.limit(10).includes(:latest_comments)

posts.each do |post|
  post.latest_comments.each do |comment|
    puts comment.body
  end
end

Now you can pick one

Now you know 5 difference ways to solve the problem, and you can pick the one that better fits your needs =)

If you are not sure, I think you can use the first one, an object to fetch all and select just the latest N default order, unless your metrics are telling you that you need something different.

More about window functions

Here are some useful articles if you want to learn more about window functions and how to use them in rails.

Window functions - If you want to learn how they work in postgres, this is a link to the docs.
ActiveRecord Eager Loading with Query Objects and Decorators - A post from Thoughtbot with a proposal to use query objects, decorators and window functions to model this problem.
Demo: “Last N of each” query with PostgreSQL and Rails ActiveRecord - A demo from vovimayhem to help you learn how to translate the SQL for the window functions to Arel and Rails.

More about lateral join

Here are some useful articles if you want to learn more about the lateral join in postgres.

Lateral queries - If you want to learn how they work in postgres, this is a link to the docs.
Group by limit per group in PostgreSQL - A post that help me understand how you can use the lateral join in a “top-n-per-group” query
Use lateral join when finding latest City status - A pull request with a refactor from window functions to lateral join

10 Examples to learn when your ActiveRecord query will be executed

2020-12-08T23:29:00Z

Do you struggle fixing N+1 queries because is hard for you to detect why ActiveRecord seems to ignore your “includes”?

Wouldn’t it be nice to just see some piece of code and know when the query will execute?

Well… this exercise will try to help you achieve this… to help you identify when a query will execute by just watching the code :)

Instructions

I will show you 10 code examples using ActiveRecord’s query interface, like includes, order and where.

For each code example you will:

Try to identify when ActiveRecord will execute de query
Compare your answer with the one I will provide after the example.

Warning!

If you are just starting, the excercises could be hard!… Don’t worry, that’s expected, with practice it will be easier!

Excercise 1

Post.includes(:comments).to_a

Answer

When calling to_a

Excercise 2

Post.includes(:comments).order(:id).to_a

Answer

When calling to_a

Excercise 3

Post.all.each do |post|
  post.comments
  post.sorted_comments.to_a
end

Answer

When calling each on Post.all.each
When calling to_a on post.sorted_comments

Excercise 4

Post.includes(:comments).each do |post|
  post.comments.to_a
end

Answer

When calling each on Post.includes(:comments).each

Excercise 5

Post.includes(:comments).each do |post|
  post.comments.order(:id)
  post.comments.order(:id).to_a
end

Answer

When calling each on Post.includes(:comments).each
When calling to_a on post.comments.order(:id).to_a

Excercise 6

Post.includes(:sorted_comments).each do |post|
  post.sorted_comments.to_a
  post.comments.to_a
end

Answer

When calling each on Post.includes(:sorted_comments).each
When calling to_a on post.comments.to_a

Excercise 7

Post.includes(:sorted_comments).each do |post|
  post.sorted_comments.first
  post.comments.first
end

Answer

When calling each on Post.includes(:sorted_comments).each
When calling first on post.comments.first

Excercise 8

Post.includes(:sorted_comments).map do |post|
  post.sorted_comments.last
  post.comments.last
end

Answer

When calling map on Post.includes(:sorted_comments).map
When calling last on post.comments.last

Excercise 9

Post.includes(:sorted_comments).sort_by(&:id).each do |post|
  post.sorted_comments.limit(1).first
  post.comments.to_a.limit(1).first
end

Answer

When calling sort_by on Post.includes(:sorted_comments).sort_by(&:id).each
When calling first on post.sorted_comments.limit(1).first
When calling first on post.comments.to_a.limit(1)

Excercise 10

Post.includes(:comments).where("comments.points > 5").references(:comments).each do |post|
  post.comments.to_a
  post.comments.order(:id).to_a
  post.comments.sort_by(&:id).to_a
end

Answer

When calling each on Post.includes(:comments).where("comments.points > 5").references(:comments).each
When calling to_a on post.comments.order(:id).to_a

Practice more…

Been able to identify when a query will execute, just by watching the code is an ability that will make you job, much more easy.

Maybe at the beginning it will be hard, but the more you practice the easier that it will be.

So if you already can answer all the excercises right, try to create your own excercises and practice until you feel confident of your ability.

Other thing that you could do is to actually run the code of each excercise and also experiment changing them to try to understand what’s happening. You can find the code on bhserna/when the query is executed

Why Active Record seems to ignore your "includes" and runs a query for each record?

2020-11-20T13:26:00Z

Do you have a method that works fine with a single record, but when you use it on a list causes N+1 queries?

Imagine that you have a model like this…

class Posts
  has_many :comments

  def latest_comment
    comments.order(:created_at).last
  end
end

When you are fetching a single post there is no problem…

post = Post.find(id)
puts post.latest_comment

But when you try to fetch a list of posts, the method seems to ignore the includes, and runs a query for each post to get the latest_comment for each post!

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

Why???

I think that the first step, to fix this kind of problems, is to understand how the method chaining in Active Record works.

Method chaining in Active Record

Active Record implements “method chaining” which allow us tu use multiple Active Record methods together.

You can chain methods in a statement when the previous method called returns an ActiveRecord::Relation, like all, where, includes, joins and order.

You can’t chain Active Record methods, after a call to a method that does not return an ActiveRecord::Relation, like to_a, find or last. You need to put those methods at the end of the statement.

Note: You can learn more on the rails guides

What is the problem in the example?

When you are fetching a single post there is no problem, because all the methods before the call to last return an ActiveRecord::Relation

post = Post.find(id)
puts post.latest_comment

If you try each link of the chain, you will see that just the call to last does not return an ActiveRecord::Relation.

relation = ActiveRecord::Relation
comments = Post.find(id).comments

puts comments.is_a?(relation) #=> true
puts comments.order(:id).is_a?(relation) #=> true
puts comments.order(:id).last.is_a?(relation) #=> false

But when you try to fetch the list…

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

At the moment you call each on the ActiveRecord::Relation, it will execute the query, and in your logs you will see something like this…

Post Load (0.2ms)  SELECT "posts".* FROM "posts"
Comment Load (0.4ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ORDER BY "comments"."id" ASC  [["post_id", 2001], ["post_id", 2002], ["post_id", 2003], ["post_id", 2004], ["post_id", 2005], ["post_id", 2006], ["post_id", 2007], ["post_id", 2008], ["post_id", 2009], ["post_id", 2010]]

But then for each post it will execute a new query, because although you already have the comments loaded, with order(:created_at).last you are building a new query to fetch the latest comment with a different order.

That is the reason why when you try to fetch a list of posts, the method seems to ignore the includes, and runs a query for each post to get the latest_comment for each post.

How can you solve this problem?

Maybe your problem is not exactly like the example, but if you are also trying to fetch the latest “X” in a list of records I think that you can use an association with a default order.

Something like this…

class Post
  has_many :comments, -> { order(:created_at) }

  def latest_comment
    comments.last
  end
end

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

It won’t execute n+1 queries, because you are fetching the comments already on the right order, and you don’t need to ask the database to sort them again.

If you want to see other options you can see this post with some other ways to solve the problem

Do you want to practice?

Now that you understand when the query will be execute, maybe you will want to do a little excercise to practice this ability and try to make it automatic.

You can try this little excercise with 10 examples to help you learn when your ActiveRecord query will be executed.

PDF with the benchmarks for the fixes to the latest-comment n+1 problem

2020-11-19T22:19:00Z

When you are trying to solve an n+1 problem, hopefully you will find different ways to solve it. But what is the one that you should use?

Talking specifically about the fixes that I share to you to solve the “latest comment” n+1 queries problem… I think you can use the first one, “the association with default order”, unless your metrics are telling you that you need something different.

But any way, I think that been aware of the performance implications of your code and understanding the tradeoffs is a good thing…

That’s why I want to share with you some benchmarks to the 5 fixes in different scenarios, to help you understand which solution is better in each case, and which solution would be enough for your case.

I have already shared the benchmarks for this 5 fixes on html format.

But now I put the report also on PDF =)

Tool to run the benchmark for fixes for the latest-comment n+1 queries problem

2020-11-14T12:57:00Z

What fix should you pick for your “latest-comment” n+1 queries problem?

One example of this problem is trying to get the latest comment on a list of posts, but there are others, like the last review in a list of products, or the cheapest price, or the latest payment for each costumer, etc…

I have already shared some fixes to help you solve the problem. and some benchmarks for this fixes….

This is the code that I use to run those benchmarks. You can use it to play with the code and seed values, to make it more specific to your current problem.

Benchmark structure

The benchmark will include the memory and the “iterations per second” benchmarks.

With this “reports"…

sorted_comments - Default order for the "has many” association
latest_comment - A “has one” association for the latest comment
feed looping - Looping through the latest comment for each post
cached feed - A version of the looping where we cache feed in memory
russian_doll - Russian doll-caching (with a simpler cache mechanism)
cached_comment - Caching the latest comment in a column

The link

You can see the code on github at bhserna/latest_comment_n-1_fixes_benchmark

Cheatsheet to help you decide which fix for the last-comment n+1 problem is the best for you

2020-11-13T02:38:00Z

Do you have to fetch the latest “x” of each record? Not sure how to avoid the n+1 queries?

Hopefully you will find different ways to solve it. But what is the one that you should use?

You can search the internet, but opinions can be overwhelming and confusing, and most of the time not specific to your specific usecase.

I have already shared some fixes that to solve the “latest comment” n+1 queries problem and some benchmarks for this 5 fixes…

Now I want to share with you a little cheatsheet to help you decide which fix to pick in different contexts, based mostly on my interpretation of the benchmarks and the complexity of each solution.

I hope it can help you =)

Tool to run the fixes for the latest-comment n+1 queries problem

2020-11-10T12:56:00Z

Do you have to fetch the latest “x” of each record?

I want to share with you the code I used in the post 5 ways to fix the latest-comment n+1 problem.

You can use it to run the examples and play with the code and seed values, to pick the right solution for you current case.

You can see the code on github at bhserna/latest_comment_n-1_examples

Benchmarks for the fixes to the latest-comment n+1 problem

2020-11-02T14:19:00Z

What fix should you pick for your n+1 queries problem?

When you are trying to solve an n+1 problem, hopefully you will find different ways to solve it. But what is the one that you should use?

You can search the internet, but opinions can be overwhelming and confusing, because… “Window functions are great option here"… or "Window functions will work, but they can be horrible on performance"… "I’d go with caching (in a column)"… "You should use Russian-doll caching"… "I’d write custom SQL"…

Talking specifically about the fixes that I share to you to solve the "latest comment” n+1 queries problem… I think you can use the first one, “the association with default order”, unless your metrics are telling you that you need something different.

But any way, I think that been aware of the performance implications of your code and understanding the tradeoffs is a good thing…

Want a pdf with the report?

Get it here:
Benchmarks for the fixes to the latest-comment n+1 problem

Benchmark structure

You will have results for this scenarios…

Posts: 20, Comments: 5 per post
Posts: 20, Comments: 10 per post
Posts: 20, Comments: 50 per post
Posts: 20, Comments: 100 per post
Posts: 20, Comments: 1000 per post

Each scenario will include the memory and the “iterations per second” benchmarks.

With this “reports"…

sorted_comments - Default order for the "has many” association
latest_comment - A “has one” association for the latest comment
feed looping - Looping through the latest comment for each post
cached feed - A version of the looping where we cache feed in memory
russian_doll - Russian doll-caching (with a simpler cache mechanism)
cached_comment - Caching the latest comment in a column

All the reports where tested in a MacBook Pro (13-inch 2020) with:

Processor: 1.4 Ghz Intel Core i5, 4 cores
Memory: 8gb

Benchmark code

This is the code used for the benchmark…

require "./config"
require "./seeds"
require "benchmark"
require "benchmark/memory"
require "benchmark/ips"

class Post < ActiveRecord::Base
  has_many :comments
  has_many :sorted_comments, -> { order(:id) }, class_name: "Comment"
  has_one :latest_comment, -> { Comment.latest_comments_for_posts }, class_name: "Comment"
  belongs_to :cached_comment, class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post
  after_commit :update_cached_comment

  def self.latest_comments_for_posts
    where(id: select("max(id)").group(:post_id))
  end

  private

  def update_cached_comment
    post.update(cached_comment_id: id)
  end
end

class Feed
  def posts
    @posts ||= fetch_posts
  end

  def fetch_posts
    posts = Post.all
    comments = Comment.latest_comments_for_posts.group_by(&:post_id)
    posts.map { |post| FeedPost.new(post, comments[post.id]&.first) }
  end
end

class FeedPost
  attr_reader :latest_comment

  def initialize(post, latest_comment)
    @post = post
    @latest_comment = latest_comment
  end
end

def cache(post)
  @cache ||= {}
  @cache[post.id] ||= yield
end

def clear_cache!
  @cache = {}
end

Seeds.run(posts_count: 20, comments_count: 1000)

n = 1
puts ""
puts "Memory Benchmark"
puts "---------"

Benchmark.memory do |benchmark|
  benchmark.report("sorted_comments") do
    n.times do
      Post.includes(:sorted_comments).all.each do |post|
        post.sorted_comments.last.body
      end
    end
  end

  benchmark.report("latest_comment") do
    n.times do
      Post.includes(:latest_comment).all.each do |post|
        post.latest_comment.body
      end
    end
  end

  benchmark.report("feed looping") do
    n.times do
      Feed.new.posts.each do |post|
        post.latest_comment.body
      end
    end
  end

  feed = Feed.new
  benchmark.report("cached feed") do
    n.times do
      feed.posts.each do |post|
        post.latest_comment.body
      end
    end
  end

  benchmark.report("cached_comment") do
    n.times do
      Post.includes(:cached_comment).all.each do |post|
        post.cached_comment.body
      end
    end
  end

  benchmark.report("russian doll") do
    n.times do
      Post.all.each do |post|
        cache(post) do
          post.latest_comment.body
        end
      end
    end
  end

  clear_cache!
  benchmark.compare!
end


puts ""
puts "Benchmark ips"
puts "---------"


Benchmark.ips do |benchmark|
  benchmark.config(:time => 5, :warmup => 1)

  benchmark.report("sorted_comments") do
    Post.includes(:sorted_comments).all.each do |post|
      post.sorted_comments.last.body
    end
  end

  benchmark.report("latest_comment") do
    Post.includes(:latest_comment).all.each do |post|
      post.latest_comment.body
    end
  end

  benchmark.report("feed looping") do
    Feed.new.posts.each do |post|
      post.latest_comment.body
    end
  end

  feed = Feed.new
  benchmark.report("cached feed") do
    n.times do
      feed.posts.each do |post|
        post.latest_comment.body
      end
    end
  end

  benchmark.report("cached_comment") do
    Post.includes(:cached_comment).all.each do |post|
      post.cached_comment.body
    end
  end

  benchmark.report("russian doll") do
    Post.all.each do |post|
      cache(post) do
        post.latest_comment.body
      end
    end
  end

  benchmark.compare!
end

You can see the code on github at bhserna/latest_comment_n-1_fixes_benchmark

Benchmark results

Now here you have the reports…

Scenario 1: Posts: 20, Comments: 5 per post

Memory Benchmark
---------
Calculating -------------------------------------
     sorted_comments   484.348k memsize (    15.950k retained)
                         5.630k objects (   146.000  retained)
                        50.000  strings (    50.000  retained)
      latest_comment   116.855k memsize (     2.065k retained)
                         1.426k objects (    16.000  retained)
                        50.000  strings (    15.000  retained)
        feed looping    55.176k memsize (     0.000  retained)
                       702.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
         cached feed    55.136k memsize (    35.840k retained)
                       701.000  objects (   390.000  retained)
                        50.000  strings (    45.000  retained)
      cached_comment    70.032k memsize (    40.000  retained)
                       883.000  objects (     1.000  retained)
                        50.000  strings (     1.000  retained)
        russian doll   488.744k memsize (     1.304k retained)
                         7.808k objects (    24.000  retained)
                        50.000  strings (    20.000  retained)

Comparison:
         cached feed:      55136 allocated
        feed looping:      55176 allocated - 1.00x more
      cached_comment:      70032 allocated - 1.27x more
      latest_comment:     116855 allocated - 2.12x more
     sorted_comments:     484348 allocated - 8.78x more
        russian doll:     488744 allocated - 8.86x more

Benchmark ips
---------
Warming up --------------------------------------
     sorted_comments    29.000  i/100ms
      latest_comment    58.000  i/100ms
        feed looping    92.000  i/100ms
         cached feed    14.814k i/100ms
      cached_comment    76.000  i/100ms
        russian doll   324.000  i/100ms
Calculating -------------------------------------
     sorted_comments    317.872  (± 4.4%) i/s -      1.595k in   5.026951s
      latest_comment    606.529  (± 4.1%) i/s -      3.074k in   5.077027s
        feed looping      1.020k (± 3.4%) i/s -      5.152k in   5.056148s
         cached feed    153.459k (± 2.5%) i/s -    770.328k in   5.022960s
      cached_comment    803.337  (± 2.7%) i/s -      4.028k in   5.017891s
        russian doll      3.322k (± 2.1%) i/s -     16.848k in   5.073899s

Comparison:
         cached feed:   153459.3 i/s
        russian doll:     3322.0 i/s - 46.19x  (± 0.00) slower
        feed looping:     1020.1 i/s - 150.43x  (± 0.00) slower
      cached_comment:      803.3 i/s - 191.03x  (± 0.00) slower
      latest_comment:      606.5 i/s - 253.01x  (± 0.00) slower
     sorted_comments:      317.9 i/s - 482.77x  (± 0.00) slower

Scenario 2: Posts: 20, Comments: 10 per post

Memory Benchmark
---------
Calculating -------------------------------------
     sorted_comments   586.444k memsize (    15.950k retained)
                         6.930k objects (   146.000  retained)
                        50.000  strings (    50.000  retained)
      latest_comment   116.855k memsize (     2.065k retained)
                         1.426k objects (    16.000  retained)
                        50.000  strings (    15.000  retained)
        feed looping    55.176k memsize (     0.000  retained)
                       702.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
         cached feed    55.136k memsize (    35.840k retained)
                       701.000  objects (   390.000  retained)
                        50.000  strings (    45.000  retained)
      cached_comment    70.032k memsize (    40.000  retained)
                       883.000  objects (     1.000  retained)
                        50.000  strings (     1.000  retained)
        russian doll   488.744k memsize (     1.304k retained)
                         7.808k objects (    24.000  retained)
                        50.000  strings (    20.000  retained)

Comparison:
         cached feed:      55136 allocated
        feed looping:      55176 allocated - 1.00x more
      cached_comment:      70032 allocated - 1.27x more
      latest_comment:     116855 allocated - 2.12x more
        russian doll:     488744 allocated - 8.86x more
     sorted_comments:     586444 allocated - 10.64x more

Benchmark ips
---------
Warming up --------------------------------------
     sorted_comments    25.000  i/100ms
      latest_comment    52.000  i/100ms
        feed looping    95.000  i/100ms
         cached feed    15.271k i/100ms
      cached_comment    78.000  i/100ms
        russian doll   325.000  i/100ms
Calculating -------------------------------------
     sorted_comments    240.019  (± 5.0%) i/s -      1.200k in   5.011152s
      latest_comment    532.712  (± 4.1%) i/s -      2.704k in   5.085499s
        feed looping    987.484  (± 2.8%) i/s -      4.940k in   5.006529s
         cached feed    156.682k (± 1.7%) i/s -    794.092k in   5.069703s
      cached_comment    744.650  (± 7.4%) i/s -      3.744k in   5.059516s
        russian doll      3.135k (± 4.9%) i/s -     15.925k in   5.092878s

Comparison:
         cached feed:   156681.7 i/s
        russian doll:     3134.7 i/s - 49.98x  (± 0.00) slower
        feed looping:      987.5 i/s - 158.67x  (± 0.00) slower
      cached_comment:      744.7 i/s - 210.41x  (± 0.00) slower
      latest_comment:      532.7 i/s - 294.12x  (± 0.00) slower
     sorted_comments:      240.0 i/s - 652.79x  (± 0.00) slower

Scenario 3: Posts: 20, Comments: 50 per post

Memory Benchmark
---------
Calculating -------------------------------------
     sorted_comments     1.408M memsize (    15.950k retained)
                        17.330k objects (   146.000  retained)
                        50.000  strings (    50.000  retained)
      latest_comment   116.855k memsize (     2.065k retained)
                         1.426k objects (    16.000  retained)
                        50.000  strings (    15.000  retained)
        feed looping    55.176k memsize (     0.000  retained)
                       702.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
         cached feed    55.136k memsize (    35.840k retained)
                       701.000  objects (   390.000  retained)
                        50.000  strings (    45.000  retained)
      cached_comment    70.032k memsize (    40.000  retained)
                       883.000  objects (     1.000  retained)
                        50.000  strings (     1.000  retained)
        russian doll   488.744k memsize (     1.304k retained)
                         7.808k objects (    24.000  retained)
                        50.000  strings (    20.000  retained)

Comparison:
         cached feed:      55136 allocated
        feed looping:      55176 allocated - 1.00x more
      cached_comment:      70032 allocated - 1.27x more
      latest_comment:     116855 allocated - 2.12x more
        russian doll:     488744 allocated - 8.86x more
     sorted_comments:    1408084 allocated - 25.54x more

Benchmark ips
---------
Warming up --------------------------------------
     sorted_comments     8.000  i/100ms
      latest_comment    29.000  i/100ms
        feed looping    92.000  i/100ms
         cached feed    15.385k i/100ms
      cached_comment    76.000  i/100ms
        russian doll   343.000  i/100ms
Calculating -------------------------------------
     sorted_comments     87.033  (± 3.4%) i/s -    440.000  in   5.060406s
      latest_comment    287.003  (± 3.8%) i/s -      1.450k in   5.060856s
        feed looping    877.671  (± 3.0%) i/s -      4.416k in   5.036340s
         cached feed    152.899k (± 2.3%) i/s -    769.250k in   5.033712s
      cached_comment    719.988  (± 5.6%) i/s -      3.648k in   5.084290s
        russian doll      3.284k (± 6.0%) i/s -     16.464k in   5.035281s

Comparison:
         cached feed:   152899.4 i/s
        russian doll:     3284.4 i/s - 46.55x  (± 0.00) slower
        feed looping:      877.7 i/s - 174.21x  (± 0.00) slower
      cached_comment:      720.0 i/s - 212.36x  (± 0.00) slower
      latest_comment:      287.0 i/s - 532.75x  (± 0.00) slower
     sorted_comments:       87.0 i/s - 1756.80x  (± 0.00) slower

Scenario 4: Posts: 20, Comments: 100 per post

Memory Benchmark
---------
Calculating -------------------------------------
     sorted_comments     2.434M memsize (    15.950k retained)
                        30.330k objects (   146.000  retained)
                        50.000  strings (    50.000  retained)
      latest_comment   116.855k memsize (     2.065k retained)
                         1.426k objects (    16.000  retained)
                        50.000  strings (    15.000  retained)
        feed looping    55.176k memsize (     0.000  retained)
                       702.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
         cached feed    55.136k memsize (    35.840k retained)
                       701.000  objects (   390.000  retained)
                        50.000  strings (    45.000  retained)
      cached_comment    70.032k memsize (    40.000  retained)
                       883.000  objects (     1.000  retained)
                        50.000  strings (     1.000  retained)
        russian doll   488.744k memsize (     1.304k retained)
                         7.808k objects (    24.000  retained)
                        50.000  strings (    20.000  retained)

Comparison:
         cached feed:      55136 allocated
        feed looping:      55176 allocated - 1.00x more
      cached_comment:      70032 allocated - 1.27x more
      latest_comment:     116855 allocated - 2.12x more
        russian doll:     488744 allocated - 8.86x more
     sorted_comments:    2433508 allocated - 44.14x more

Benchmark ips
---------
Warming up --------------------------------------
     sorted_comments     4.000  i/100ms
      latest_comment    19.000  i/100ms
        feed looping    79.000  i/100ms
         cached feed    15.535k i/100ms
      cached_comment    85.000  i/100ms
        russian doll   353.000  i/100ms
Calculating -------------------------------------
     sorted_comments     47.354  (± 4.2%) i/s -    240.000  in   5.075750s
      latest_comment    185.682  (± 2.2%) i/s -    931.000  in   5.016461s
        feed looping    754.015  (± 1.7%) i/s -      3.792k in   5.030455s
         cached feed    154.707k (± 1.1%) i/s -    776.750k in   5.021351s
      cached_comment    819.179  (± 2.0%) i/s -      4.165k in   5.086336s
        russian doll      3.390k (± 2.6%) i/s -     17.297k in   5.104940s

Comparison:
         cached feed:   154707.3 i/s
        russian doll:     3390.5 i/s - 45.63x  (± 0.00) slower
      cached_comment:      819.2 i/s - 188.86x  (± 0.00) slower
        feed looping:      754.0 i/s - 205.18x  (± 0.00) slower
      latest_comment:      185.7 i/s - 833.18x  (± 0.00) slower
     sorted_comments:       47.4 i/s - 3267.07x  (± 0.00) slower

Scenario 5: Posts: 20, Comments: 1000 per post

Memory Benchmark
---------
Calculating -------------------------------------
     sorted_comments    20.977M memsize (    15.950k retained)
                       264.330k objects (   146.000  retained)
                        50.000  strings (    50.000  retained)
      latest_comment   116.855k memsize (     2.065k retained)
                         1.426k objects (    16.000  retained)
                        50.000  strings (    15.000  retained)
        feed looping    55.176k memsize (     0.000  retained)
                       702.000  objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
         cached feed    55.136k memsize (    35.840k retained)
                       701.000  objects (   390.000  retained)
                        50.000  strings (    45.000  retained)
      cached_comment    70.032k memsize (    40.000  retained)
                       883.000  objects (     1.000  retained)
                        50.000  strings (     1.000  retained)
        russian doll   488.744k memsize (     1.304k retained)
                         7.808k objects (    24.000  retained)
                        50.000  strings (    20.000  retained)

Comparison:
         cached feed:      55136 allocated
        feed looping:      55176 allocated - 1.00x more
      cached_comment:      70032 allocated - 1.27x more
      latest_comment:     116855 allocated - 2.12x more
        russian doll:     488744 allocated - 8.86x more
     sorted_comments:   20977100 allocated - 380.46x more

Benchmark ips
---------
Warming up --------------------------------------
     sorted_comments     1.000  i/100ms
      latest_comment    22.000  i/100ms
        feed looping    25.000  i/100ms
         cached feed    14.818k i/100ms
      cached_comment    57.000  i/100ms
        russian doll   288.000  i/100ms
Calculating -------------------------------------
     sorted_comments      4.665  (± 0.0%) i/s -     24.000  in   5.177711s
      latest_comment    206.056  (± 7.3%) i/s -      1.034k in   5.050651s
        feed looping    239.214  (± 4.2%) i/s -      1.200k in   5.026543s
         cached feed    146.296k (± 5.1%) i/s -    740.900k in   5.079435s
      cached_comment    706.969  (±10.8%) i/s -      3.534k in   5.064015s
        russian doll      3.235k (± 4.9%) i/s -     16.416k in   5.087702s

Comparison:
         cached feed:   146296.1 i/s
        russian doll:     3235.0 i/s - 45.22x  (± 0.00) slower
      cached_comment:      707.0 i/s - 206.93x  (± 0.00) slower
        feed looping:      239.2 i/s - 611.57x  (± 0.00) slower
      latest_comment:      206.1 i/s - 709.98x  (± 0.00) slower
     sorted_comments:        4.7 i/s - 31362.27x  (± 0.00) slower

So… What fix should you pick?

I think that you can use the simplest solution, “the association with default order”, maybe until you are in the 50 comments per post, after that maybe you could try other options like “fragment caching” or “looping through the latest comment for each post”.

Download a cheatsheet

You can also download a cheatsheet to help you decide which fix to pick in different contexts, based mostly on my interpretation of the benchmarks and the complexity of each solution.

Get it here: Cheatsheet for the “latest-comment” n+1 queries problem

5 ways to fix the latest-comment n+1 problem

2020-10-20T22:32:00Z

Do you have a method that works fine with a single record, but when you use it on a list causes N+1 queries?

One example of this problem is trying to get the latest comment on a list of posts, but there are others, like the last review in a list of products, or the cheapest price, etc…

But let’s explain the problem with…

The latest comment on a list of posts

Imagine, that you have a model like this…

class Posts
  has_many :comments

  def latest_comment
    comments.order(:created_at).last
  end
end

When you are fetching a single post there is no problem…

post = Post.find(id)
puts post.latest_comment

But when you try to fetch a list of posts, the method seems to ignore the includes and runs a query for each post to get the latest_comment for each post!

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

You can write a different method to get the latest_comment with ruby (not ActiveRecord), but it feels inelegant, and maybe it can be confusing later…

def self.latest_comment(post)
  post.comments.sort_by(&:created_at).last
end

The good news is that there are other solutions =)… And I will share with you some of them. So, you can select the one that is better for the current state of your app.

1. A default order for the “has_many” association

Instead of sorting the comments on the latest_comment method, you can sort the comments at the association level and just ask for the last sorted comment.

class Post
  has_many :sorted_comments, -> { order(:created_at) }, class_name: "Comment"

  def latest_comment
    sorted_comments.last
  end
end

Post.includes(:sorted_comments).each do |post|
  puts post.latest_comment
end

2. A “has_one” association for the latest comment

Instead of loading all the comments for all the posts, you can use a has_one association for the latest_comment, providing a scope with just the latest_comment for each post.

class Post < ActiveRecord::Base
  has_many :comments
  has_one :latest_comment, -> { Comment.latest_comments_for_posts }, class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post

  def self.latest_comments_for_posts
    latest_comments_ids = select("max(id)").group(:post_id)
    where(id: latest_comments_ids)
  end
end

Post.includes(:latest_comment).all.each do |post|
  puts post.latest_comment.body
end

3. Looping through the latest comment for each post

You can query the database for a list with the latest_comment for each post, group them by post_id and then pick the right comment for each post.

You can also use some specific objects the task, If you want to avoid polluting you Post and Comment models.

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post

  def self.latest_comments_for_posts
    latest_comments_ids = select("max(id)").group(:post_id)
    where(id: latest_comments_ids)
  end
end

class Feed
  def posts
    posts = Post.all
    comments = Comment.latest_comments_for_posts.group_by(&:post_id)
    posts.map { |post| FeedPost.new(post, comments[post.id]&.first) }
  end
end

class FeedPost
  attr_reader :latest_comment

  def initialize(post, latest_comment)
    @post = post
    @latest_comment = latest_comment
  end
end

feed = Feed.new
feed.posts.map do |post|
  puts post.latest_comment.body
end

4. Russian doll-caching

If you are already using fragment cache on your app, you can embrace the “Russian doll caching” and think of n+1 as a feature, and let the n+1 queries run just the first time and use the cache after that.

class Post < ActiveRecord::Base
  has_many :comments

  def latest_comment
    comments.order(:id).last
  end
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

In your view…

<% #posts/index.html.erb %>
<%= render partial: 'posts/post', collection: @posts, cached: true %>

<% #posts/_post.html.erb %>
<% cache post %>
  <%= post.title %>
  <%= post.latest_comment.body %>
<% end %>

5. Caching the latest comment

You can also add a new column for the last_comment_id and a belongs_to association, and keep the last comment in sync “manually” (maybe with an after_commit callback).

ActiveRecord::Schema.define do
  create_table :posts do |t|
    t.column :title, :string
    t.column :body, :text
    t.column :latest_comment_id, :integer
  end

  create_table :comments do |t|
    t.column :body, :text
    t.column :post_id, :integer
  end
end

class Post < ActiveRecord::Base
  has_many :comments
  belongs_to :latest_comment, class_name: "Comment"
end

class Comment < ActiveRecord::Base
  belongs_to :post
  after_commit :update_latest_comment

  private

  def update_latest_comment
    post.update(latest_comment_id: id)
  end
end

Post.includes(:latest_comment).all.each do |post|
  puts post.latest_comment.body
end

Other options for the “latest comments for posts” query

You can try other options for the query. I used one that worked on postgres and sqlite3, but maybe you want to try other depending on your database.

This work on sqlite3 and postgres (used in the examples)…

def self.latest_comments_for_posts
  latest_comments_ids = select("max(id)").group(:post_id)
  where(id: latest_comments_ids)
end

This works on postgres…

def self.latest_comments_for_posts
  from(order(:post_id, id: :desc).arel.distinct_on(arel_table[:post_id]).as("comments"))
end

And this works on sqlite3…

def self.latest_comments_for_posts
  from(order(id: :desc), "comments").group(:post_id)
end

Do you want to see some benchmarks?

If you want to see how this fixes behave in different scenarios, you can take a look to the benchmarks for this 5 fixes.

Or play with the code for those benchmarks. You can see the code on github at bhserna/latest_comment_n-1_fixes_benchmark

Do you want to run the examples?

You can see the code on github at bhserna/latest_comment_n-1_examples.

You can use it to run the examples and play with the code and seed values, to pick the right solution for you current case.

Now you can pick one

Now you know 5 difference ways to solve the problem, and you can pick the one that better fits your app =)

If you are not sure, I think you can use the first one, “the association with default order”, unless your metrics are telling you that you need something different.

Download a cheatsheet

You can also download a cheatsheet to help you decide which fix to pick in different contexts, based mostly on my interpretation of the benchmarks and the complexity of each solution.

Get it here: Cheatsheet for the “latest-comment” n+1 queries problem

Create a new controller for the hidden abstraction and reduce conditionals

2020-10-06T22:51:00Z

Have you ever implemented a feature that does work, but has resulted in using multiple conditionals in your controller and views… Wondering if there is a better way to do what you just did?

Maybe your problem is that you have a hidden abstraction and you need a new controller…

Do you have a hidden abstraction?

Not always, but some times, you will be able to simplify your code by creating a new abstraction… and a lot of the times you already know the name of it, and you are calling it by its name, but is not expressed in the code.

For example imagine that you are building an app that lets you create “Invoices”, and you want to add the possibility to create “Estimates"… but those estimates are not a new kind of record they are "just simple invoice records”, but with an estimate boolean attribute set to true.

Something like this…

class Invoice < ActiveRecord::Base
  has_many :line_items
end

invoice = Invoice.new
estimate = Invoice.new(estimate: true)

And to implement the feature you will have, in you views… a “New invoice” button and a “New estimate” button and both send you to the invoice controller like this (not a big trouble yet)…

<%= link_to "New invoice", new_invoice_path %>
<%= link_to "New estimate", new_invoice_path(estimate: true) %>

In the controller something like this…

def new
  if params[:estimate] == "true"
    @invoice = Invoice.new(estimate: true)
  else
    @invoice = Invoice.new
  end
end

And on your other views (show, form, and mailer views)…. You will have this all over the place…

<%= if @invoice.estimate? %>
  <h1>Estimate</h1>
<% else %>
  <h1>Invoice</h1>
<% end %>

<%= if @invoice.estimate? %>
  <%= link_to "Cancel", invoices_path(estimate: true) %>
<% else %>
  <%= link_to "Cancel", invoices_path %>
<% end %>

Is this your case?

If this is your case, then create a new controller

If you already are have a name for this thing, “Estimates”, you already have an abstraction, and maybe the first step that you can do is to create a new controller for it…

For example in this case you could have…

A “New estimate” button to a new dedicated route…

resources :invoices
resources :estimates

<%= link_to "New estimate", new_estimate_path %>

In the controller you will need just one object, and you will be able to use the expected name…

def new
  @estimate = Invoice.new(estimate: true)
end

And for you other views (show, form, and mailer views)…. You won’t need the conditionals any more…

<h1>Estimate</h1>

<%= link_to "Cancel", estimates_path %>

Maybe you can think that you are repeating to much… but normally duplication is far cheaper than the wrong abstraction.

Give it a try =)

Association with default order, to fix the latest-comment n+1 problem

2020-09-30T12:19:00Z

Do you have a method that works fine with a single record, but when you use it on a list causes N+1 queries?

Maybe, you are building a rails app and you want to fetch a list of posts with the latest comment (or something similar)…

You have a model like this…

class Posts
  has_many :comments

  def latest_comment
    comments.order(:created_at).last
  end
end

When you are fetching a single post there is no problem…

post = Post.find(id)
puts post.latest_comment

But when you try to fetch a list of posts, the method seems to ignore the includes and runs a query for each post to get the latest comment for each post =(.

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

You can write a different method to get the latest comment with ruby (and not ActiveRecord), but it feels weird, and you know that it can bring you troubles later =(.

def self.latest_comment(post)
  post.comments.sort_by(&:created_at).last
end

Is this the only solution?

Well, nop… there are many ways to solve this =)

And although ther is no perfect solution, I want to tell you the one that I think that you should try first…

A default order for the association

On the definition of your association, you will pass a scope with the default order. It looks like this…

class Post
  has_many :comments, -> { order(:created_at) }

  def latest_comment
    comments.last
  end
end

Post.includes(:comments).each do |post|
  puts post.latest_comment
end

If you are not sure about the default order in the association, you can create a new association…

class Post
  has_many :comments
  has_many :sorted_comments, -> { order(:created_at) }, class_name: “Comment”

  def latest_comment
    sorted_comments.last
  end
end

Post.includes(:sorted_comments).each do |post|
  puts post.latest_comment
end

If you want to know more about how this part of the has_many association work, you can check with the “Scopes” section in the docs.

I hope this tip, can help you =)

Note: As as I said, this solution is not perfect, it can cause some performance problems if you have many comments per posts, because you are preloading all the coments for the list of posts, but don’t worry for now unless you will have many (maybe more than 50) comments per post =)

Want to know more ways to solve this problem?

If you want to review other options, you can see this post with other 4 ways to fix this problem.

Benito Serna

Fix n+1 queries on rails

Fix n+1 queries on Rails

Making things work isn't enough for you any more?

Do you feel unqualified to tackle complex tasks?

Maybe you know the basics...

Do you worry when they ask you for more?

But, what if you could...

Learn how to fix n+1 queries on rails

Table of contents

No more… “Why active record is ignoring my includes?”

Use preloading to avoid n+1 queries

Count without n+1 queries

Save counts with counter caches

Caching custom counts and other computed values

"Please help me with my n+1 problem..."

“Is my fix right?”

“I know how to fix the problem!”

Undestand the basics to avoid n+1 queries

Start solving n+1 queries problems with total confidence!

MdRecord: File-backed models for Rails

Why?

Installation

Basic Usage

Custom Path

Categorized Content

Rails Integration

Rendering to HTML

Links

Fixing a bug in my lateral joins queries with rails

The problem

Proposed solution

A new tool to define top-N-per-group associations using lateral joins

Simple Searchable module for searching with Rails and SQLite's LIKE

The Searchable Module

How to Use It

The tests to show how each part works

Maybe you can use it

A "read-more" behavior truncating by number of lines with css line-clamp and Stimulus.js

The example

The html and css

The javascript

Other ways to do it…

Copy shoes.rb and use stacks and flows to build layouts

What are a stacks and a flows?

Personal testing of the idea

Maybe this names can break your UI kit

Fix n+1 queries by caching computed values

Some examples of possible expensive computations

Should you always cache computations?

Patterns to save computed values

Using a before_save callback

Using after_touch callback

Using after_commit + background job

Be aware of race conditions

Tools to help you detect n+1 queries

Rails strict_loading

How it can help you?

References

Rack mini profiler

How it can help you?

How does the report looks like?

References

Bullet

How can it help you?

How does the error report looks like?

References

Prosopite

How can it help you?

How does the error report looks like?

References

NPlusOneControl

How can it help you?

How does the error report looks like?

References

Truncate in the middle with truncate rails helper

The omission parameter

A form with two buttons with formation and formmethod

Changing the action with formaction

Changing the method with formmethod

Changing the action with `formaction`

Changing the method with `formmethod`