Rails add Turbo / Hotwire to existing sprockets application

Yesterday I've added turbo/hotwire to a big rails application which evolved over several years from rails 3, to 4 to 5 to 6 and now to rails 7.

The 6-7 version updates went pretty smooth. But now we would like to rewrite our javascript-coffeescript kruft to a more modern approach.

The app is very big, so directly replacing everything is virtually impossible. That's why I introduced an incremental approach.

The 7 version is still running sprockets. (Glad I didn't introduce webpacker in version 6, which would have resulted in an even bigger kruft).

New situation

The new situation will keep sprockets for the current javascript/coffeescript, sass css and other assets.

It will introduce esbuild for building the new javascript.

The directory structure used is the following:

  • app/assets/builds: is the output folder of esbuild javascripts
  • app/assets/javascripts: contains the legacy scripts
  • app/javascript: contains the new javascripts

Changes in Gemfile

- gem uglifier
+ gem terser 

+ gem turbo-rails
+ jsbundling-rails
+ stimulus-rails

Configure sprockets / deployment

Sprockets needs to to include the esbuild build directory to embed the new javascript content.
Changes in app/assets/config/manifest.js

+//= link_tree ../build/

Add the build directory to the assets paths in config/initializers/assets.rb

Rails.application.config.assets.paths << Rails.root.join('app/assets/builds')

Uglifier was crashing on production deployment of the esbuild javascript files (those are already handled). This was solved be replacing uglifier with terser in my Gemfile.

This also needs to be enabled in the config/environments/production.rb file.

- config.assets.js_compressor = Uglifier.new(harmony: true)
+ config.assets.js_compressor = :terser

Required legacy javascript changes

Every legacy javascript file that uses the ready eventhandler is replaced by the turbo:load event

- $(document).ready ->
+$(document).on "turbo:load", ->

New javacript changes

The new new javascript files cannot be called application.js because it has the same name as the legacy name. Because the same name is generated to solve this, I use app/javascript/app.js:

// Entry point for the build script in your package.json

import { Turbo } from "@hotwired/turbo-rails"
window.Turbo = Turbo

import "./controllers"

Contents of app/javascript/controllers/index.js

// This file is auto-generated by ./bin/rails stimulus:manifest:update
// Run that command whenever you add a new controller or create them with
// ./bin/rails generate stimulus controllerName

import { application } from "./application"

import LegacyJsController from "./legacy_js_controller"
application.register("legacy-js", LegacyJsController)

Load the new javascripts

Add the new javascript include to the layout template of the project and `views/layouts/application.html.erb'

= <%= javascript_include_tag "application", defer: true %>
+ <%= javascript_include_tag "app", defer: true %>

- <body>
+ <body 'data-turbo'= <%= @turbo ? true : false %>' >

Because Turbo requires 422 status code on invalid form result, I've disabled Turbo by default. I've tried enabling it by default, which worked pretty good except for form-validation errors. And there are a LOT of places this happen, so for the incremental update it's better to slowly convert/change all pages for using turbo.

When a controller-action uses turbo it can set the @turbo variable to true.
The idea it to slowly introduce this to every controller. When I'm confident it works (almost) everywhere this can be inverted/removed.

Add hotwire/turbo to the package.json file

Contents of package.json

{
 "name": "projectname",
 "private": true,
 "dependencies": {
   "@hotwired/stimulus": "^3.0",
   "@hotwired/turbo-rails": "^7.3.0",
   "esbuild": "^0.17.11"
 },
 "scripts": {
   "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --public-path=assets"
 },
 "version": "0.1.0"
}

(Build with yarn)

Running the app

Running the app now happens via bin/dev which uses Foreman so esbuild automaticly builds the new javascript:

web: unset PORT && bin/rails server
js: yarn build --watch

How it is going

Currently I'm slowly moving legacy javascript methods to stimulus controllers.
Specialised autocomplete-inputs, select2 inputs are converted ony by one to stimulus controllers.

My experience is that simulus is very good in auto-enabling inputs on loading ajax content.
Even my legacy html-updates via ajax are updated by stimulus when they are introduced into the DOM.
I really like this solution, it's possible to keep the old legacy javascript running and build new things with the turbo/stimulus approach

Rails ActiveStorage import on Mac OS X, missing files

Capistrano-db-tasks is a nice tool to pull production data to the local environment. This nicely imports all SQL data and imports the activestorage files from the /storage folders to the development environment.

When importing the active storage files, not all blob-files are imported. So you get missing file on your local machine. Which is not very desirable, certainly when you are working on a rails upgrade and want to test if everything is working.

I'm using a MacOS X development machine. By default OS X uses a case insensitive filesystem (APFS).

The production machine uses FreeBSD or Linux as OS. Which has a case sensitive filesystem.

ActiveStorage generate case sensitive filenames by default (Why!?!). So you can have a folder 'xE' and 'Xe', which are 2 different folders on BSD/Linux, but not on my Mac.

Today I configured a solution suggested by Brian Boyko: https://brianboyko.medium.com/a-case-sensitive-src-folder-for-mac-programmers-176cc82a3830

Mounting a case-sensitive container. '/Volumes/casesensitive'

Via disutility you can create a case-sensitive APFS container. This container shares the space of you main disk. But is case-sensitive.

I use this container to link my storage folders. (I don't have the need to have a case sensitive code directory) .
This is extra nice, because the gigantic storage files aren't stored in my projects folder anymore.

ln -s /Volumes/casesensitive/project/storage ~/project/storage

Airbrake / errbit error reporting not working with puma 6 / rails 7

(Note issue below has been resolve, see link below for best resolution)

Last few weeks I busy updating several of my Ruby on Rails apps to Ruby 3.2.0 and Rails 7.0. (running Puma 6)
Because upgrading an app requires me to monitor the apps, and get notified when something happens, it's crucial that my error notification tool works.
I use airbrake with errbit as backend.

Most apps report perfectly after upgrade. But I notice one of my Apps didn't report any errors.

This app used the preload! functionality of puma.
With the preload option:

  • rake airbrake:test just works.
  • Exception from sidekiq Jobs are reported,
  • but controller exceptions aren't working

Removing the preload! of puma solve the issue.


But this isn't always possible, in one of my apps, I use the embedded puma variant. In this article Mike describes the steps how to do this.

This method requires a preload. (so removing it isn' a good solution)

After a lot of debugging/testing I found a simple workaround to solves the non-reporting issue. Forcing a syncronized notify instead of the default async.
I noticed the async config option was removed from airbrake, so I don't know how to disable it. For now I use the flexibility of Ruby to change the default notify implementation to the synchronized variant.

## force to airbrake to be NOT async, to solve issues with preload! puma
def Airbrake.notify(...)
  Airbrake.notify_sync(...)
end

With this included in my airbrake initializer the errors are noticed again :-)

NOTE
I've created a PR for ruby/timeout to fix this issue https://github.com/ruby/timeout/pull/25

Which has been merged in the v0.3.2 release of the Timeout gem. 🎉
https://github.com/ruby/timeout/releases/tag/v0.3.2

Incompatibility rails 5.2 and mail gem: ArgumentError (:arguments expected to be an Array of individual string args)

After an innocent gem security update, suddenly my application didn't mail anymore. :S
The following crash happend when the rails application tried to send an e-mail.

Error performing ActionMailer::DeliveryJob (Job ID: xxx) from Async(mailers) in 17.17ms: ArgumentError (:arguments expected to be an Array of individual string args):
gems/mail-2.8.0/lib/mail/network/delivery_methods/sendmail.rb:53:in `initialize'
gems/mail-2.8.0/lib/mail/message.rb:278:in `new'
gems/mail-2.8.0/lib/mail/message.rb:278:in `delivery_method'
gems/actionmailer-5.2.8.1/lib/action_mailer/delivery_methods.rb:65:in `wrap_delivery_behavior'
gems/actionmailer-5.2.8.1/lib/action_mailer/delivery_methods.rb:79:in `wrap_delivery_behavior!'
gems/actionmailer-5.2.8.1/lib/action_mailer/base.rb:823:in `mail'

After digging deeper in the exception trace and git diff, I figured out the mail gem was upgraded from 2.7.1 to 2.8.0.

The sendmail default sendmail arguments configuration value given by actionmailer is "-i". Which was fine for the older mail gem.

The new gem requires this to be an list of strings.
You can downgrade this OR use the following workaround in your environment file (config/environments/production.rb)

 config.action_mailer.sendmail_settings = {
   location:  "/usr/sbin/sendmail", arguments: ["-i"] 
}

Btw: It's a known issue and probably resolved in the next 2.8.1 release (https://github.com/mikel/mail/issues/1541)

PHP fastcgi logging with woo-commerce / WordPress

We have a synchronisation script running with the REST API from WooCommerce. This script is running for years. But suddenly it received gateway errrors

In the /var/log/nginx/error.log, the following error messages appeared

2022/08/03 22:41:19 [error] 1233#33321: *323108 upstream sent too big header while reading response header from upstream, client: 10.0.0.1, server: www.example.com, request: "POST /wp-json/wc/v3/products/1234/variations/batch HTTP/1.1", upstream: "fastcgi://unix:/tmp/example.com.sock:", host: "www.example.com"

It seems PHP is sending debug information over the FASTCGI headers.
This isn't desirable.

To fix it, disable it in php.ini. (it's on by default. WHY!?)

fastcgi.logging = 0

Btw. these errors/warnings are from low quality wordpress-plugins. (which cannot be replaced directly)