Development
GCP Pub/Sub with Ruby on Rails
Streamline cross-component communication in your micro-service architecture
TL;DR Checkout Cloudenvoy, a Ruby gem relying on Google Cloud Pub/Sub to greatly simplify cross-component communication.
The purpose of a micro-service or lambda architecture is to break down your platform logic into smaller reusable components which are more maintainable from a code and scalability perspective.
But as you build this type of architecture a few questions start popping up:
- How should these components talk to each other?
- How to ensure data consistency between these components?
- Should they talk to the same database?
- Should they replicate the data?
Should you share the same table/database between components to have a shared context?
In a world where Kubernetes, GCP Cloud Run or AWS Fargate exist, scaling runtimes has never been easier. What remains difficult however is scaling databases, especially relational ones.
Therefore I do not recommend sharing databases between your components as a mean to share contextual data between them because that will make your overall platform hard to scale.
You may expose private APIs on your components that can be queried by other components, but that will make your data flows complicated and introduce a lot of querying logic in your components which will bloat the code.
Instead, make your components listen to data streams and extract/replicate the data in their own datastore. This way you can address scalability problems step by step as they arise.
How should components talk to each other?
Our approach at Keypup is the following:
- Functions returning a result should be invoked via API.
- Context (contextual data and events) should be shared via streams of data. Using APIs is not a viable solution because it would require each component to call N APIs every time data must be propagated.
Cross-component communication via API is a fairly straightforward. Use any API standard (jsonapi, GraphQL, custom) and stick to it so that components can invoke functions from other components.
One question remains: how to share/stream contextual data to your components.
There are two big solutions commonly used nowadays:
- Kafka, a streaming platform originally developed by LinkedIn. Its core concept is that data should be commited in a big log (similar to git commits) and then distributed to topics that subscribers listen to. Because a "git log" is maintained, historical data can always replayed. I recommend reading the blog article from Jay Kreps on why they made the decision to create Kafka. It's an eye opener on data management.
- Pub/Sub, a pattern/implementation that focuses mainly on data distribution (publish and subscribe). GCP Pub/Sub and AWS SNS are example implementations.
In the end the concept is mostly the same for both solutions:
- Your components publish data to topics (e.g. users topic, blog posts topic, user action topic)
- Your components subscribe to topics and react to new data (store them, send an email etc.)
So today I'll write about how to leverage Pub/Sub on Google Cloud Platform to maintain data consistency across your Rails applications.
Introducing Cloudenvoy, GCP Pub/Sub the Rails way
We at Keypup are heavy users of GCP Pub/Sub and we needed a way to make message publishing and subscribing simple in our applications. So we created Cloudenvoy to do just that.
Cloudenvoy provides Ruby bindings to:
- Format and publish messages to Pub/Sub
- Receive and process messages via webhook from Pub/Sub
This is what a publisher looks like:
This is what a subscriber looks like:
A real-life example: sharing users between components
Let's say you have two Rails applications. The first app is your main platform where users signup. The second one is a communication app sending messages to your users using the most appropriate channel (email, slack, other).
The communication app is invoked by API by the main app every time a message must be sent to a user. Only the user ID is passed as part of that API invocation. This means the communication app must be able to lookup the user context to send the message via the right channel.
Setting up Cloudenvoy
First add the gem to both apps and run bundle install:
Then add an initializer to configure Cloudenvoy on each app:
We'll be working locally. So the best is to download the gcloud CLI and install the pub/sub emulator:
That's it. Let's jump into the fun part now.
Setup the publisher in the main app
What we want is publish user attributes every time a user is created/updated/deleted. We'll assume your main app has an ActiveRecord model called User.
Let's create a user publisher. The publisher is responsible for specifying the topic it publishes to as well as formatting the actual message payload:
Now let's modify our User model to call our UserPublisher via callbacks:
Finally ensure the topic is created on pub/sub:
That's it! User updates will now be published to pub/sub every time a user is created/updated/deleted.
Setup the subscriber in the communication app
What we want here is to upsert a local user every time we receive an update. We'll assume there is an ActiveRecord model called User in this app.
Let's create the subscriber:
Then ensure a subscription is registered in pub/sub:
Finally start your application to listen to pub/sub messages via webhook:
Done! Our communication app will now maintain its list of platform users automatically.
Playing with our new streaming flow
Start a rails console on your main app, create a user and display its id. The action of creating the user will publish an update via pub/sub:
Then start a rails console on your communication app and verify your user is there:
Feel free to update your user in the main app and verify that the user in the second app has been properly updated. I guarantee it works :)
> Pub/Sub all the data
Cloudenvoy never gets more complicated than the example above. Format -> Publish -> Receive -> Process. That's it.
With Cloudenvoy in hand you can start getting rid of all these API endpoints and ruby methods you were using to keep your Rails applications in sync with each other and simplify your data flows a LOT. Data, events, asynchronous actions...you can do them all via Pub/Sub and Cloudenvoy.
Also GCP Pub/Sub will guarantee deliverability, so you don't even need to worry about error handling if one of your components is down. Cross-component communication cannot get any simpler than that.
