FreeBSD Update all php packages to a new version

When PHP is installed on a FreeBSD system, you have a lot of packages, for example:

pkg prime-list
php80-ctype
php80-curl
php80-exif
...
php80-tokenizer
php80-zip
php80-zlib

To update all these packages to newer version the following can be used :-)

# first note what was installed (optional but recommended)
pkg prime-list > /tmp/installed-packeges.txt

# then execute the update command php80 => php83
cat /tmp/installed-packages.txt | grep php80 | sed 's/php80/php83/g' | xargs -o  pkg install

Freebsd bastille console not working after upgrade 13 to 14

After updating my host system to FreeBSD 14.0, it was time to update the bsatille jails.
The jails I've running are thin jails , with symlinks to the release of the given jail.

Summary of the update

The jail have been updated following the instructions found at the bastille manual

# ensure the new version is bootstrapped and update to the latest patch release: 
bastille bootstrap 14.0-RELEASE update

# and change the 13.2 to 14.0 mount
bastille stop TARGET
bsatille edit TARGET fstab

# force reinstallation and upgrade all packages
# the pkg boostrap  isn't in the manual but IS required
bastille start TARGET
bastille pkg TARGET bootstrap -f
bastille pkg TARGET update
bastille pkg TARGET upgrade -f
bastille restart TARGET

bastille console fails

Next the jail seemed to be running, but unfortunately bastille console TARGET just stops without any errors/warning etc.

Though bastille console didn't work, It was still possible to enter the jail via jexec

jexec TARGET

Incorrect /etc configuration files

The problem is described in the following post: https://forums.freebsd.org/threads/newbie-upgrade-problem-13-2-release-14-0-release-merge-conflict.91219/

The reason is that pam_opie.so seems to have been removed. it can be fixed by simply moving that auth method as describe there.

A much better way is to update the outdated /etc configuration files! (These haven't been updated yet!)

Update the /etc configuration with etcupdate

First make sure the freebsd sources are available in /usr/jails/bastille/releases/14.0-RELEASE/usr/src

I solved this by copying the sources from my host system to the release.

cp -Rp /usr/src/ /usr/local/bastille/releases/14.0-RELEASE/usr/src/

You can also download the source directly via this link: )
https://cgit.freebsd.org/src/snapshot/releng/14.0.tar.gz

Next perform the etcupdate in the jail.

jexec TARGET
etcupdate
etcupdate resolve # if there are conflicts

Fix merge conflicts if they happen.
After that bastille console should work again!

FreeBSD update 13 to 14

I've just updated my server from 13.2-RELEASE to 14.0-RELEASE. This when pretty flawlessly, I had an issue with slow updating of the userland part. (The update after the first reboot)

It was extremely slow, update userland was already running for more then an hour. Pretting Ctrl+t to view what it was doing, i noticed it was [wait]ing most of the time.

Searching for this problem I found a solution speed it up dramaticly.
Setting the following control variable (in another terminal) made the update finish almost instantly.

sysctl vfs.zfs.dmu_offset_next_sync=0

After the next reboot this flag is turned back to the original value.

Thanks to the contributors of the following references:

ActiveStorage SVG Variant Transformer

In a project I'm working on (My Hero's Journey), we use a lot SVG's for the artwork of the application. The application sometimes needs a full colored image of the SVG. Other times it needs change a certain accent color. We would love to use the ActivateStorage variant support for this. It would be a good use case for these images. Just generate it once, store it and return the different variations.

The original image. (In this image the islands are colored full red (#FF0000 with transparency))

A variant of this is image is to replace the red color (#FF0000) with green.

Or changing the hue to a given color.

The problem

Active Storage supports variants for images. Though this doesn't include SVG's, because it's a vector image instead of an rasterized image.

Under the hood the ActivateStorage uses a transformer class. ActiveStorage::Transformers::Transformer

When you look at the code, at first glance it looks like it should be able to support multiple transformers. At the moment using a different transformer is not (yet) available in ActiveStorage.

The cause of this, is the definition of the transformer in the
ActiveStorage::Variation class. It's always the ImageProcessingTransformer:

def transformer
    ActiveStorage::Transformers::ImageProcessingTransformer.new(transformations.except(:format))
end

My Solution for Now

A workaround for not being able to change the transformer class, can be done by patching the ActiveStorage::Variation#tranformer method. Make sure the following class is loaded by an initializer.

lib/active_storage/variation_custom_formatter_extension.rb`

module ActiveStorage::VariationCustomFormatterExtension
  def transformer
    return ActiveStorage::SvgTransformer.new(transformations.except(:format)) if content_type == "image/svg+xml"
    super
  end

  ActiveStorage::Variation.prepend(self)
end

Just invoke the constant on initialization to make Zeitwork load it.

config/initializers/activestorage_custom_formatter.rb

ActiveSupport::Reloader.to_prepare do
  ActiveStorage::VariationCustomFormatterExtension
end

A general implementation could to add a factory, which creates a tranformer based on a given mime-type.

To use SVG's as images in ActiveStorage, and allow variations the following configuration options are required.

config/initializers/active_storage_svg_support.rb

# SVG shouldn't be served as binary.
Rails.application.config.active_storage.content_types_to_serve_as_binary -= ['image/svg+xml']

# Make the default content-disposition of the svg's inline. (just like another image)
Rails.application.config.active_storage.content_types_allowed_inline += ['image/svg+xml']

# Make the svg content type 'variable', so variations are enabled
Rails.application.config.active_storage.variable_content_types += ['image/svg+xml']

ActiveStorage Transformer

An ActiveStorage transformer in rails uses a hash with transformations that are applied to the given image.
For example, a variant of rasterized image can be defined by passing several operations/settings:

image_tag asset.file.variant(
  resize_to_limit: [100, 100],
  format: :jpeg,
  saver: { subsample_mode: "on", strip: true, interlace: true, quality: 80 })

A simple SVG transformer

We could also define all kinds operations for SVG files. In our application I've defined two operations (for now). replace_color, to replace a color by another, and set_hue, to change the hue of an image.

image_tag asset.file.variant(set_hue: "#FF0000", format: :svg)

Below is the implementation of this transformer class. (It uses the color_conversion gem, to perform rgb to hsl calculations)

The main entry point that's being called by ActiveStorage is process. This method is called with two arguments, file and format. The file is a temporary file which contains the original data. When you don't rewind the tempfile, you get ActiveStorage::IntegrityErrors. Because ActiveStorage requires the file to be open at the first byte in the file.

The transformations applied to this formatter can be accessed via the method transformations.

lib/active_storage/svg_transformer.rb

class ActiveStorage::SvgTransformer < ActiveStorage::Transformers::Transformer
  def process(file, format:) # :doc:
    content = file.read
    transformations.each do |name, data|
      case name
      when :set_hue then content = set_hue(content, data)
      when :replace_color then content = replace_color(content, data)
      else raise "SVG Transformation not supported #{name}"
      end
    end

    out = Tempfile.new(["svg_transformer", ".svg"])
    out.write(content)
    out.rewind
    out
  end

  # { hue: #EE00AA }
  private def set_hue(content, color_in)
    hex_color_regexp = /#[A-F0-9]{3,8}/i
    color_tone = ColorConversion::Color.new(color_in).hsl
    content = content.gsub(hex_color_regexp) do |hex_in|
      hex, alpha = extract_alpha(hex_in)
      color = ColorConversion::Color.new(hex).hsl
      new_color = ColorConversion::Color.new(h: color_tone[:h], s: color[:s], l: color[:l])
      "#{new_color.hex}#{alpha}"
    end
    content
  end

  # Extract the hex and alpha part of a hex color
  # - #RGBA => #RGB, AA
  # - #RRGGBBAA => #RRGGBB, AA
  private def extract_alpha(hexa)
    case hexa.length
    when 5 then [hexa[0..3], hexa[4] * 2]
    when 9 then [hexa[0..6], hexa[7..8]]
    else [hexa, ""]
    end
  end

  # { replace_color: { "#FF0000" => "#4455aa" , ... } }
  private def replace_colors(content, key_value_map)
    key_value_map.each do |key, value|
      content = content.gsub(/#{key}/i, value)
    end
    content
  end
end

The SVG images are drawn with Affinity Designer and exported with hex color codes. At the moment the colors of the images are simply adjusted via a regexp replace. (Full XML parsing/building could be used for complex operations)

And that's all!

I'm pretty happy with the solution. The usage of variants makes use of the groundwork of ActiveStorage handling. Which generates it just once and store it on disk. And automaticly deletes the variants when the main image is deleted.

It would be very nice if ActiveStorage added support for multiple Variant transformers for different content-type. Audio transformers, PDF transformers, CSV transformers, or docx variations. (There is a related PR request about this feature. But it seems to be stalled)

Btw, I'm aware you can do very much with SVG on the browser side, like CSS coloring and JS dynamics. This works very well when rending an SVG inline in an HTML file. When using external SVG's as images this becomes a lot harder!

Turbo Frame Reload Error: “element has a source URL which references itself”

I love Marco's turbo_power library for adding some extra actions to turbo. For example the turbo_frame_reload which should reload a given turbo frame.

But the turbo_frame_reload didn't work in my situation.

I've got an index view which renders all orders in a turbo-frame (with pagination).
Pressing a button next to an order should tag the order and reload the orders page.
Tagging is an action in other controller, which doesn't know anything about this order screen. But after tagging it should refresh the turbo frame with the orders.

I thought this could be easy by refreshing the turbo_frame with a turbo_stream action turbo_frame_reload. This only works if the frame has a src attribute in it.

When I set src on the this frame I get the error: element has a source URL which references itself. So this isn't working.

Workaround I created for now is the following.

I add an extra turbo_stream action named: turbo_frame_reload_with_data. This sets the source on the attribute data-src when the src attribute is empty/missing. (that's on the initial load).

So define the turboframe like this:

<%= turbo_frame_tag :orders, data_src: request.original_url do %>
.. content .. 
<% end %>

To make this work the following code is added to the application.js, adding a custom turbo-stream action.
This action sets the src of the turbo-frame when a turbo_frame_reload_with_data action is invoked

Turbo.StreamActions.turbo_frame_reload_with_data = function() {
  this.targetElements.forEach((element) => {
    let src = element.getAttribute('src') || element.getAttribute('data-src')
    element.src = ''
    element.src = src
  })
}

Note: Always setting src with data-src doesn't work. The data-src isn't updated on turbo-frame updates, only the src-flag is.

To make the rails experience nicer, a helper is added to the turbo_stream tag builder.

module TurboStreamActionsHelper
  def turbo_frame_reload_with_data(target = nil, **attributes)
  custom_action :turbo_frame_reload_with_data, target:, attributes:
  end
end
Turbo::Streams::TagBuilder.prepend(TurboStreamActionsHelper)

In the controller performing the action, I render the following turbo-stream action. To refresh after this, the following can be returend.

render turbo_stream: turbo_stream.turbo_frame_reload_with_data('orders'),

References