Broadcasting real-time database changes on a budget
While building Becky's (yet unreleased) app for her strength training business, I've been taking liberal advantage of the Hotwire combo of Turbo and Stimulus to facilitate dynamic frontend behavior without resorting to writing separate server-side and client-side apps. You can technically use these without Rails, but let's be honest: few people do.
Here are a few capabilities this broader suite of libraries give you, in case you're not familiar:
- Rails Request.js offers a minimal
API for sending conventional HTTP requests from JavaScript with the headers Rails
expects like
X-CSRF-Tokenhandled for you - Turbo streams can send just a snippet of HTML over the wire (a fetch/XHR or an Action Cable socket) to re-render part of a page, and support was recently added for Custom Actions that sorta let you send anything you want over the wire
- The turbo-rails gem adds some very handy glue code to broadcast model updates in your database and render Turbo streams to subscribers via an Action Cable socket connection
- Stimulus values are synced with the DOM as data attributes on the owning controller's element, and Object serialization is supported (as one might guess) via JSON serialization. Stimulus controllers, by design, don't do much but they do watch the DOM for changes to their values' data attributes
Is your head spinning yet? Good.
Here are the design constraints I was up against last week:
- Display a full-screen, highly interactive workout page that's relatively expensive for the backend to build (translating a program to a series of blocks and sets for the user, assembling all the supporting video assets, etc.)
- As workouts are marked "complete" or "skipped" by the user, it needs to survive page refreshes, so that means it has to store that state somewhere. (Other tiny bits of state cropped up, too, like a record of which movements had been substituted and a notes field for each set)
- Because a user can only "be in" one workout at a time, visiting the Workout tab on multiple devices needed to show the same workout and reflect the same progress
- If the user completes a set on one device, that should be reflected on all the user's other devices without requiring them to refresh the page (lest they be confused by being told on their iPad to do a set they already completed on their phone)
- I am an essentialist (a fancy word for cheapskate). I don't want the app to generate hundreds of database rows every time someone does a workout, or make a dozen queries every time the user clicks on something, or send large superfluous chunks of HTML over the wire unnecessarily, or try to fix any such waste by introducing a caching layer
That's the thing about web sites that looks like apps! People have come to expect native app-like experiences. And something that most native apps at least try to do is silently sync data between devices in the background.
To accomplish this, here's what I did:
- Store a user's progress, movement substitutions, and notes as JSONB
columns
in a single
WorkoutLogrecord, so as to create only one row for each workout by each user ⬇ - On first render, those JSONB columns are set as values on the page's primary
stimulus controller via data attributes. When the user clicks a button (like
"Complete" on a set), the
progressvalue is updated. And whenever a value changes, an HTTP patch request with the value is sent to a one-off controller action that updates just that column of just that record ⬇ - Thanks to an after_update_commit
hook
on the
WorkoutLog, check the saved_changes hash for changes to those JSONB columns. If they changed, call broadcast_render_to or broadcast_render_later_to to render those changes to a one-off Turbo stream partial template ⬇ - In that template, invoke a custom Turbo stream action I wrote that doesn't render HTML templates at all, but rather updates the JSONB columns' analogous data attributes on the primary controller's bound element ⬇
- Wire it up by connecting to the same Action Cable subscription from the workout page via its view template ⬇
Is this a lot? Yes. There are a lot of moving parts. There's a lot of context-switching to set it up. These features are new and the documentation isn't always fantastic, either.
But also, the first time you see this work end-to-end, it's hard not to have a holy fucking shit moment. Then, when you see how small a diff was needed to achieve such a cool feature, it will inspire a wow, this is goddam awesome reaction. Having two browser windows side-by-side and watching as changes made in one are instantaneously reflected in the other feels magical.
Below I'll include some snippets to help you get started with something similar, mapping to each of the 5 steps above
Setting up the model
Nothing too extraordinary here. Just one JSONB and two JSONB[] columns:
class AddStateToWorkoutLogs < ActiveRecord::Migration[7.2]
def change
change_table :workout_logs do |t|
t.jsonb :progress, default: {}, null: false
t.jsonb :movement_substitutions, array: true, default: [], null: false
t.jsonb :experiences, array: true, default: [], null: false,
end
end
end
And the model:
class WorkoutLog < ApplicationRecord
belongs_to :user
# attribute configuration, validation, etc…
end
In the model itself, I actually built custom Rails attributes for better handling small JSON objects with symbolized keys in Postgres array columns, and then an entire JSON object validation framework, but that's a story for another day. What you see above is all that was needed for the real-time broadcasting functionality we're discussing right now.
Handling JSON object changes in Stimulus
Here are the relevant bits on the element the page's primary Stimulus controller
is bound to. We'll just focus on the progress JSONB column here:
<%= content_tag :div,
id: "workout_player",
data: {
controller: "workout-player",
workout_player_progress_value: @workout_log.progress.to_json,
workout_player_update_progress_url_value: update_workout_log_progress_path(@workout_log),
action: "turbo:after-update-dataset->workout-player#streamUpdatedPlayerState",
} do %>
<!-- HTML -->
<% end %>
Note the DOM ID (so the Turbo stream can easily target it), the data
attributes for the current progress value as JSON as well as the URL path
(generated by a Rails URL helper), and my custom turbo:after-update-dataset
event (which you'll see fired later) bound as an action on the controller.
And here are the relevant bits of the Stimulus controller itself:
// app/javascript/controllers/workout_player_controller.js
import { Controller } from '@hotwired/stimulus'
import { patch } from '@rails/request.js'
export default class extends Controller {
static values = {
progress: {
type: Object,
default: {}
},
updateProgressUrl: String,
}
// Called whenever the user takes an action that changes the progress Object
uploadProgress () {
patch(this.updateProgressUrlValue, {
body: { progress: this.progressValue }
})
}
progressValueChanged () {
// Fired after the data attribute is mutated by the stream
}
streamUpdatedPlayerState () {
// Does stuff specific to stream updates.
// Will be bound to turbo:after-update-dataset
}
}
See comments above for what each method does, but right now, imagine that
clicking a "Complete" button triggered an action that in turn called
uploadProgress to send an HTTP patch request to the server.
Broadcasting changes to subscribed clients
To handle the patch, I wrote a simple fire-and-forget action to persist the change:
class WorkoutLogsController < ApplicationController
def update_progress
workout_log = current_user.workout_logs.find(params[:id])
if workout_log.update(progress: params[:progress])
head :no_content
else
head :unprocessable_entity
end
end
end
And here's where the magic happens! This is the full after_update_commit hook
on WorkoutLog for all three of the separately-updatable JSONB columns:
class WorkoutLog < ApplicationRecord
after_update_commit -> {
if [:progress, :finished_at].any? { |attr| saved_changes.key?(attr) }
broadcast_render_to(user_id, id, :workout_player,
partial: "workout_logs/progress_update")
end
if saved_changes.key?(:movement_substitutions)
broadcast_render_to(user_id, id, :workout_player,
partial: "workout_logs/movement_substitutions_update")
end
if saved_changes.key?(:experiences)
broadcast_render_to(user_id, id, :workout_player,
partial: "workout_logs/experiences_update")
end
}
end
The docs encourage you to favor the later variant (e.g.
broadcast_render_later_to instead of broadcast_render_to) broadcast methods
to kick them off to a job queue, because otherwise sending them will occupy a
web worker's time. However, in this case, none of these partials require so much
as a database query in order to transmit a single HTML element, so it's probably
less work overall to just fire them off in the same request-response lifecycle.
Also, note the first several arguments to broadcast_render_to: the user's ID,
the workout log's ID, and the name :workout_player. The broadcast methods
take a variable number of arguments used to construct a subscription identifier
for Action Cable to broadcast to.
This is important! If we had merely identified the subscription as :workout_player:
broadcast_render_to(:workout_player,
partial: "workout_logs/movement_substitutions_update")
Then every update to every workout by every user would be broadcast to
everyone all the time. Instead, we call broadcast_render_to(user, id, :workout_player) to uniquely scope these broadcasts to that user's particular
workout (i.e. "browsers looking at the same workout log for this user").
By the way, I've enjoyed using the actioncable-enhanced-postgresql-adapter gem to managing my Action Cable subscriptions in Postgres, avoiding introducing a Redis dependency to my production environment.
Rendering a custom Turbo stream action
Here's the "progress_update" partial we identified above:
<!-- app/views/workout_logs/_progress_update.turbo_stream.erb -->
<%= tag.turbo_stream(action: :update_dataset, target: "workout_player",
data: {
workout_player_progress_value: workout_log.progress.to_json,
workout_player_workout_finished_elsewhere_value: workout_log.finished?,
}) %>
This partial will render a <turbo-stream> tag specifying an as-yet-undefined
Turbo stream action named update_dataset and targeting the DOM node with ID
workout_player. The gobbledy-gook sent over each subscriber's Action Cable
connection will looks like this:
{
"identifier": "{\"channel\":\"Turbo::StreamsChannel\",\"signed_stream_name\":\"IloyxGtPaTh2WjNKdlp5OVZjMlZ5THpJOjMzMjp3b3Jrb3V0X3BsYXllciI=--d49ded58c4f9fd9d53209443e5bdba92f7ea7d8afc289e313bf098f7a2eed320\"}",
"message": "<turbo-stream action=\"update_dataset\" target=\"workout_player\" data-workout-player-progress-value=\"{"blocks":[{"name":"Warm-up","sets":[{"status":"complete"}]},{"name":"Squat","sets":[{"status":"complete"},{"status":"skipped"},{"status":"complete"},{"status":"skipped"},{"status":"complete"},{"status":"complete"}]},{"name":"Combo","sets":[{"status":"incomplete"}]},{"name":"Combo","sets":[{"status":"complete"},{"status":"complete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"}]},{"name":"Combo","sets":[{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"},{"status":"incomplete"}]},{"name":"Finisher","sets":[{"status":"incomplete"},{"status":"incomplete"}]}]}\" data-workout-player-workout-finished-elsewhere-value=\"false\"></turbo-stream>\n"
}
It's ugly, but it's less than 2 kilobytes. (It'd be cool if Action Cable included gzip compression support, then it'd only be 596 bytes over the wire.)
Finally, we need to define that custom Turbo stream action designed for updating data attributes as opposed to rendering HTML templates. Feel free to copy and paste this next bit for your own uses:
// app/javascript/ext/turbo_rails_ext.js
import { Turbo } from '@hotwired/turbo-rails'
Turbo.StreamActions.update_dataset = function () {
const target = document.getElementById(this.getAttribute('target'))
const targets = target ? [target] : document.querySelectorAll(this.getAttribute('targets'))
if (targets.length === 0) return
targets.forEach(target => {
target.dispatchEvent(new CustomEvent('turbo:before-update-dataset', { detail: { element: target } }))
for (const [key, val] of Object.entries(Object.assign({}, this.dataset))) {
target.dataset[key] = val
}
target.dispatchEvent(new CustomEvent('turbo:after-update-dataset', { detail: { element: target } }))
})
}
Subscribe to the Action Cable from the view
Finally, the coup de grâce! All that's left is to set up a subscription to
Action Cable matching the one we're broadcasting to. We can do this with the
turbo_stream_from helper provided by turbo-rails in the view:
<%= turbo_stream_from(current_user, @workout_log.id, :workout_player) %>
Note that we're using the same tuple (user ID, workout log ID, and the symbol
:workout_player) as we passed to broadcast_render_to to connect the client
with the correct messages from the server.
That's it!
This stuff demos really well, so it really was an act of self-restraint on my part not to spend all day putting together an interactive demo and a full sample codebase along with this post.
What do you think? Want to see this in action? Then come to my Rails World talk next month! Until then, you'll just have to use your imagination as to how cool this stuff is.