Airbrake / errbit error reporting not working with puma 6 / rails 7

(Note issue below has been resolve, see link below for best resolution)

Last few weeks I busy updating several of my Ruby on Rails apps to Ruby 3.2.0 and Rails 7.0. (running Puma 6)
Because upgrading an app requires me to monitor the apps, and get notified when something happens, it's crucial that my error notification tool works.
I use airbrake with errbit as backend.

Most apps report perfectly after upgrade. But I notice one of my Apps didn't report any errors.

This app used the preload! functionality of puma.
With the preload option:

  • rake airbrake:test just works.
  • Exception from sidekiq Jobs are reported,
  • but controller exceptions aren't working

Removing the preload! of puma solve the issue.


But this isn't always possible, in one of my apps, I use the embedded puma variant. In this article Mike describes the steps how to do this.

This method requires a preload. (so removing it isn' a good solution)

After a lot of debugging/testing I found a simple workaround to solves the non-reporting issue. Forcing a syncronized notify instead of the default async.
I noticed the async config option was removed from airbrake, so I don't know how to disable it. For now I use the flexibility of Ruby to change the default notify implementation to the synchronized variant.

## force to airbrake to be NOT async, to solve issues with preload! puma
def Airbrake.notify(...)
  Airbrake.notify_sync(...)
end

With this included in my airbrake initializer the errors are noticed again :-)

NOTE
I've created a PR for ruby/timeout to fix this issue https://github.com/ruby/timeout/pull/25

Which has been merged in the v0.3.2 release of the Timeout gem. 🎉
https://github.com/ruby/timeout/releases/tag/v0.3.2

Incompatibility rails 5.2 and mail gem: ArgumentError (:arguments expected to be an Array of individual string args)

After an innocent gem security update, suddenly my application didn't mail anymore. :S
The following crash happend when the rails application tried to send an e-mail.

Error performing ActionMailer::DeliveryJob (Job ID: xxx) from Async(mailers) in 17.17ms: ArgumentError (:arguments expected to be an Array of individual string args):
gems/mail-2.8.0/lib/mail/network/delivery_methods/sendmail.rb:53:in `initialize'
gems/mail-2.8.0/lib/mail/message.rb:278:in `new'
gems/mail-2.8.0/lib/mail/message.rb:278:in `delivery_method'
gems/actionmailer-5.2.8.1/lib/action_mailer/delivery_methods.rb:65:in `wrap_delivery_behavior'
gems/actionmailer-5.2.8.1/lib/action_mailer/delivery_methods.rb:79:in `wrap_delivery_behavior!'
gems/actionmailer-5.2.8.1/lib/action_mailer/base.rb:823:in `mail'

After digging deeper in the exception trace and git diff, I figured out the mail gem was upgraded from 2.7.1 to 2.8.0.

The sendmail default sendmail arguments configuration value given by actionmailer is "-i". Which was fine for the older mail gem.

The new gem requires this to be an list of strings.
You can downgrade this OR use the following workaround in your environment file (config/environments/production.rb)

 config.action_mailer.sendmail_settings = {
   location:  "/usr/sbin/sendmail", arguments: ["-i"] 
}

Btw: It's a known issue and probably resolved in the next 2.8.1 release (https://github.com/mikel/mail/issues/1541)

PHP fastcgi logging with woo-commerce / WordPress

We have a synchronisation script running with the REST API from WooCommerce. This script is running for years. But suddenly it received gateway errrors

In the /var/log/nginx/error.log, the following error messages appeared

2022/08/03 22:41:19 [error] 1233#33321: *323108 upstream sent too big header while reading response header from upstream, client: 10.0.0.1, server: www.example.com, request: "POST /wp-json/wc/v3/products/1234/variations/batch HTTP/1.1", upstream: "fastcgi://unix:/tmp/example.com.sock:", host: "www.example.com"

It seems PHP is sending debug information over the FASTCGI headers.
This isn't desirable.

To fix it, disable it in php.ini. (it's on by default. WHY!?)

fastcgi.logging = 0

Btw. these errors/warnings are from low quality wordpress-plugins. (which cannot be replaced directly)

Security Update Rails (CVE-2022-32224)

Updating rails with the secrurity update CVE-2022-32224, "Possible RCE escalation bug with Serialized Columns in Active Record".
can cause troubles in rails projects. (I had several project that has issues with this fix).
https://discuss.rubyonrails.org/t/cve-2022-32224-possible-rce-escalation-bug-with-serialized-columns-in-active-record/81017

The main change is that the YAML loader is using safe_load to parse the string. Most classes aren't supported anymore.
Several essential classes have been removed.

To support my Spree Commerce rails projects, I've had to add the following initializer.

ActiveRecord::Base.yaml_column_permitted_classes += [BigDecimal, Symbol]

Other projects requried the HasWithIndifferentAccess

ActiveRecord::Base.yaml_column_permitted_classes += [ ActiveSupport::HashWithIndifferentAccess]

Please try to keep the number of supported classes.
I personally prefer to use JSON for new projects. Because it's simple and clean.

Derailed benchmark not running on production environment

We're running a large rails application, with a lot of request.
Since our upgrade to FreeBSD 13 the live environment is leaking memory in our ruby on rails app.
(FreeBSD 12 version of the jail didn't leak)

This application uses an NFS mount to store (a lot) of client specific files.

Running derailed_benchmark live didn't work. (Pressing ctrl+t, I noticed there were a lof of NFS locks)

In the file
derailed_benchmarks-2.1.1/lib/derailed_benchmarks/load_tasks.rb

it adds '.' to the loadpath. Which causes the production environment to complete iterate over all directories. Which 'hangs', because we have a very big active-strorage directory.

Workaround is to remove the '.' file

#    %W{ . lib test config }.each do |file|
    %W{ lib test config }.each do |file|
      $LOAD_PATH << File.expand_path(file)
    end