Rails add Turbo / Hotwire to existing sprockets application

Yesterday I've added turbo/hotwire to a big rails application which evolved over several years from rails 3, to 4 to 5 to 6 and now to rails 7.

The 6-7 version updates went pretty smooth. But now we would like to rewrite our javascript-coffeescript kruft to a more modern approach.

The app is very big, so directly replacing everything is virtually impossible. That's why I introduced an incremental approach.

The 7 version is still running sprockets. (Glad I didn't introduce webpacker in version 6, which would have resulted in an even bigger kruft).

New situation

The new situation will keep sprockets for the current javascript/coffeescript, sass css and other assets.

It will introduce esbuild for building the new javascript.

The directory structure used is the following:

app/assets/builds: is the output folder of esbuild javascripts
app/assets/javascripts: contains the legacy scripts
app/javascript: contains the new javascripts

Changes in Gemfile

- gem uglifier
+ gem terser 

+ gem turbo-rails
+ jsbundling-rails
+ stimulus-rails

Configure sprockets / deployment

Sprockets needs to to include the esbuild build directory to embed the new javascript content.
Changes in app/assets/config/manifest.js

+//= link_tree ../build/

Add the build directory to the assets paths in config/initializers/assets.rb

Rails.application.config.assets.paths << Rails.root.join('app/assets/builds')

Uglifier was crashing on production deployment of the esbuild javascript files (those are already handled). This was solved be replacing uglifier with terser in my Gemfile.

This also needs to be enabled in the config/environments/production.rb file.

- config.assets.js_compressor = Uglifier.new(harmony: true)
+ config.assets.js_compressor = :terser

Required legacy javascript changes

Every legacy javascript file that uses the ready eventhandler is replaced by the turbo:load event

- $(document).ready ->
+$(document).on "turbo:load", ->

New javacript changes

The new new javascript files cannot be called application.js because it has the same name as the legacy name. Because the same name is generated to solve this, I use app/javascript/app.js:

// Entry point for the build script in your package.json

import { Turbo } from "@hotwired/turbo-rails"
window.Turbo = Turbo

import "./controllers"

Contents of app/javascript/controllers/index.js

// This file is auto-generated by ./bin/rails stimulus:manifest:update
// Run that command whenever you add a new controller or create them with
// ./bin/rails generate stimulus controllerName

import { application } from "./application"

import LegacyJsController from "./legacy_js_controller"
application.register("legacy-js", LegacyJsController)

Load the new javascripts

Add the new javascript include to the layout template of the project and `views/layouts/application.html.erb'

= <%= javascript_include_tag "application", defer: true %>
+ <%= javascript_include_tag "app", defer: true %>

- <body>
+ <body 'data-turbo'= <%= @turbo ? true : false %>' >

Because Turbo requires 422 status code on invalid form result, I've disabled Turbo by default. I've tried enabling it by default, which worked pretty good except for form-validation errors. And there are a LOT of places this happen, so for the incremental update it's better to slowly convert/change all pages for using turbo.

When a controller-action uses turbo it can set the @turbo variable to true.
The idea it to slowly introduce this to every controller. When I'm confident it works (almost) everywhere this can be inverted/removed.

Add hotwire/turbo to the package.json file

Contents of package.json

{
 "name": "projectname",
 "private": true,
 "dependencies": {
   "@hotwired/stimulus": "^3.0",
   "@hotwired/turbo-rails": "^7.3.0",
   "esbuild": "^0.17.11"
 },
 "scripts": {
   "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --public-path=assets"
 },
 "version": "0.1.0"
}

(Build with yarn)

Running the app

Running the app now happens via bin/dev which uses Foreman so esbuild automaticly builds the new javascript:

web: unset PORT && bin/rails server
js: yarn build --watch

How it is going

Currently I'm slowly moving legacy javascript methods to stimulus controllers.
Specialised autocomplete-inputs, select2 inputs are converted ony by one to stimulus controllers.

My experience is that simulus is very good in auto-enabling inputs on loading ajax content.
Even my legacy html-updates via ajax are updated by stimulus when they are introduced into the DOM.
I really like this solution, it's possible to keep the old legacy javascript running and build new things with the turbo/stimulus approach

Install postgresql in FreeBSD jail enable sysvipc

Installen postgresql on a FreeBSD jail can result in an error when intializing the database.

 service postgresql initdb

running bootstrap script ... 2023-02-24 10:21:41.757 CET [29388] FATAL:  could not create shared memory segment: Function not implemented
2023-02-24 10:21:41.757 CET [29388] DETAIL:  Failed system call was shmget(key=37383, size=56, 03600).

By default shared memory via sysvipc is not allowed.

sysctl security.jail.sysvipc_allowed
#=> security.jail.sysvipc_allowed: 0

To solve this enable sysvipc for the given jail...
For bastille this can be added to the config file: /usr/local/bastille/jails/jailname/jail.conf

allow.sysvipc = 1;

Restart jail.

sysctl security.jail.sysvipc_allowed
#=> security.jail.sysvipc_allowed: 1

.htpasswd generation via openssl

Just do it with openssl

openssl passwd -apr1

Enter the password twice to get a hash that can be placed in a file with:

username:password_hash

Rails mysql structure.sql dump contains AUTO_INCREMENT

When rails generates a structure.sql dump for MySQL it contains the AUTO_INCREMENT value. Which is anoying because this is not something you want to happen.

CREATE TABLE `active_storage_variant_records` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `blob_id` bigint NOT NULL,
  `variation_digest` varchar(255) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `index_active_storage_variant_records_uniqueness` (`blob_id`,`variation_digest`),
  CONSTRAINT `fk_rails_993965df05` FOREIGN KEY (`blob_id`) REFERENCES `active_storage_blobs` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=471 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

Look at the part: AUTO_INCREMENT=471

After digging through the activerecord code, I saw I could sneak in a mysqldump option for the dump call. Yeah! 👍🏻

But there isn't such option 😕. It's an ancient BUG/Feature of mysql, which of course still isn't resolved in MySQL 8. https://bugs.mysql.com/bug.php?id=20786

Why would you like to dump the table structure (without data) with the AUTO_INCREMENT value!?

As a workaround it's possible to enhance the db:schema:dump task in a custom rake file (lib/tasks/remove_autoincrement_from_dump.rake).
So the AUTO_INCREMENT part is removed from it.

Rake::Task['db:schema:dump'].enhance do
  structure_sql_path = Rails.root.join("db/structure.sql")
  if File.exist?(structure_sql_path)
    sql = File.read(structure_sql_path)
    File.write(structure_sql_path, sql.gsub(/AUTO_INCREMENT=[0-9]+/, ""))
  end
end

References:

Rails generate both structure.sql and schema.rb

Default behaviour for rails is to generate a db/schema.rb when running migrations or dumping the database.

Schema.rb is great because it can be used for populating other database types and is used to populate the test database.
And I personally think it's great becaus of the vscode plugin Rails Schema which shows the database structure of your application in a sidebar (TIP, move this bar to the right panel).

The rake task db:migrate default behaviour is to invoke a db:schema:dump when migrating is done. By default this generates the schema.rb file.
This output of the generated file is used to popuplate the test-database..

But somethimes the db/schema.rb isn't good enough for popuplating the test database or using the file to generate the basic structure. Contraints are quirky.
(For example see previous article mysql json constraints).
To generate a native sql dump of the database structure, you can change the default structure to sql. Place the following line in your config/application.rb.

    config.active_record.schema_format = :sql

When enabling this dumps are generated to db/structure.sql.
But you loose the schema.rb dumps.

Here's a tip to work with structure.sql and just generate the schema.rb (for your lovely sidebar).

Create a file in lib/tasks/schema_dump.rb and enhance the dump task, so it also creates the schema.rb file.

Rake::Task['db:schema:dump'].enhance do
  File.open(Rails.root.join('db/schema.rb'), 'w') do |stream|
    ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, stream)
  end
end

NOTE: structure.sql contains yet another MySQL bug. (See next article for workaround)
Tip: use postgresql !