shootmail logo

Local-First Web Development with Replicache: Building smooth user experiences

Published : 2024-09-20
subhendu singh subhendu singh

Table of Contents

What are we building?

Demo Link: sveltekit-replicache-todo.pages.dev

Repo: https://github.com/subhendupsingh/sveltekit-replicache-todo

You probably know even before I say it. A TODO App.

Though it may not seem so, but this decision is a thoughtful one. To understand a concept that is new and a bit different from the traditional methods of reading and writing data from and to a database, I thought to start with a simple app will be more suitable where the readers and me, both can focus on Replicache concepts rather than focussing on the application logic.

Yeah, I know I sound smart now ;-)

What is local-first web development?

Forget replicache for a moment. Let’s think about how would we normally write our TODO app. Here are the tools we need:

  • Database - Supabase
  • Web framework - Next, SvelteKit, Remix etc
  • Optionally, we can separate our backend and frontend by using something like Express or Hono just for backend. I am not going to do that here.
  • CSS - Tailwind CSS (Don’t tell me tailwind css is not vanilla css)

Todo app the traditional way

Here I am just outlining the steps

  • We will define our database schema, say a todo table with following columns - id, task, created_at, updated_at, is_complete
  • We will define our CRUD operations.
  • We will make UI to add todos, mark them complete, delete and update them and will style them nicely and probably add the dark mode.

This app will work just fine. Now, imagine a use case where your database is hosted in Singapore, your application code is deployed on a server in Germany (think Hetzner).

One day, a Indie-hacker named SPS decides to pack his bags and work on his next Facebook from the a quaint village somewhere in the mountains. The internet connection is patchy. SPS decides to make an outline of his project on your TODO application. Somethings SPS might expect from your app is, it should work fast and changes he makes are not lost in case of internet fluctuation.

With your current setup, if there is an internet connection, SPS will be able to load his TODOs, but it will be quite slow because of the slow internet and multiple network round trips to reach the application server in Germany and database in Singapore. Also, in case the internet gets disconnected while he saves his TODOs, his changes will not be saved and he will have to do it again. Result, SPS might switch to someone else’s TODO app who has already read this article and implemented replicache.

Todo app the replicache way

Replicache way or the local-first way in general means the Create, Update and Delete operations are performed using simple javascript methods called mutators, and the Read is done using replicache subscriptions or query. Mutated data or mutations are always applied locally first, optimistically, which means that the changes are done locally and are reflected in the UI without informing the server yet. These mutations are then replayed on the server and the data changes in our database.

Now, even if the internet gets disconnected, replicache will try to replay he mutation on the server when it next gets connected, and it will keep on trying until the server replies with a confirmation that the mutations were applied. Hence, SPS can rest assured that his work will not be lost and all the changes are visible to him immediately because everything is read locally avoiding the network trips. This makes SPS happy. That’s the purpose of local-first web development, making SPS happy.

Why replicache?

  • Doesn’t offer separate storage, keep using your existing database.
  • You don’t need to run any separate service to use it.
  • Handles a lot automatically, exposes only a set of rules for you to implement
  • Free - well, there is pricing but not until your business earns more than $200k in ARR.
  • Has pink logo

In short, has so much to offer with very less expectations, be like replicache.

Replicache terminology

For local storage, replicache uses indexed db that is available with all modern browsers. Here are a few useful terms:

  • Mutations: Simple javascript functions that change data
  • Space: Logical separation of data. For example, in your TODO app, when SPS signs up, his user_id can be used to separate his data from the other users. So, this user_id is a space.
  • Space version: Every space (every user) has a version number that starts from 1. Everytime a mutation happens, this version is incremented.
  • Push endpoint: Is a simple API endpoint that is called by replicache automatically whenever it needs to apply a mutation on the server. Remember, mutations happen locally first and are then replayed on the server. Each mutation increases the version number of space.
  • Pull endpoint: This is also a simple API endpoint that replicache calls automatically after a configurable interval (default : 60s). In the request, it carries the last space version number that was synced and fetches all the changes that were done after this version number. It returns these changes and the updated version number to the client.
  • Query: To read data stored locally using replicache, you use replicache queries.
  • Subscriptions: You can subscribe to a query. Whenever the data related to query changes, the data is automatically received in the subscription callback, from there, you can update your UI.
  • Client: A browser tab is a client. Each client is assigned a unique ID by replicache.
  • Client group: A browser, a mobile device etc is a client group. Multiple clients (tabs) belong to one client group. Changes happening to one space, are visible to all the client groups associates with the space and hence, all the clients associated with each client group. For example, suppose SPS adds a todo from his mobile browser while hiking. Then after coming back to his, stay he opens his laptop, logs in with the same id from which he is logged in on his mobile phone. Now as explained, mobile and laptop are different client groups but they belong to the same space, hence, the new todo will be synced to his laptop too and all the tabs he opens in his laptop of mobile.

TODO app implementation, with replicache

Schema

replicache_space: For each user, there will be a replicache_space. Typically, user_id is space_id

replicache_client: Multiple clients

todo: A todo is added by a user. For each user, there is a corresponding space. Space has its version. Each todo, along with its data, maintains space_id it is associated with and the space_version it was modified under.

Generate replicache license

Run this command, it will return a random string, save it in your .env or similar

Disable ssr

Since replicache needs indexed db to run, which is only available in the browser, make sure to set ssr=false in wherever you want to use replicache. In Sveltekit, I am setting it in my +page.ts

Define mutators

Create a separate file called mutators.ts. Here we will add all our mutators.

mutators.ts

Initialise Replicache

Remember to initialise this at a place that doesn’t SSR, that runs on client side only. Here spaceId should typically be the userId of the logged in user. Push and pull URL should point to the API endpoints. For e.g. /api/push/+server.ts in SvelteKit.

Implementing push endpoint

/api/push/+server.ts

This endpoint pushes the local mutations to server, and if the mutations are applied successfully, increases the version of the space. I am listing down the steps that should be followed inside the push handler. I recommend opening this file on the side and read the code alongside the explanation here. The code has comments at each step to explain.

The entire push operation happens inside a transaction, so that if the push fails, the data is not corrupted. Here is the step by step explanation:

  • We get the current version of the space. If the space doesn’t exist, that means the user is new, we create a new space for the user with userId being the spaceId.
  • We calculate what the next space version would be when all the mutations are applied successfully i.e. simply version = version + 1.
  • We retrieve all the client Ids present in the push request. Remember, each browser tab opened on the same device or somewhere else is a separate client, and each client can push changes. So a single push request can have multiple mutations.
  • We get the lastMutationId for each client belonging to the clientGroupId we receive in the request. Each client maintains a lastMutationId that starts from 0 and increases with every mutation applied. This is a map of this shape : {”clientId1”: “lastMutationId1”, ”clientId2”: “lastMutationId2”}
  • In the last step, if a particular clientId doesn’t exist in the database, we create a new replicache_client.
  • We start applying mutations to the database. Each mutation contains id, clientID, mutation name and mutation args. Mutation name and args are the same that you define in the mutators.ts file. First we validate if the mutation id is correct and we should go ahead with the mutation or abort. We do that in two steps, first from lastMutationIds map we got in step 4, we get the lastMutationId for this particular clientId and increment it by 1, we call it expectedMutationID
  • If everything is right, we apply this particular mutation i.e. apply the changes to our database. In our case, each update or create operation in the todo table, will also contain the updated space version number with them, so that, in the next pull request, replicache knows that these entries were updated and should be updated on the local client also.
  • In the next two steps, when the mutations have been applied, we increment the lastMutationId for each client that pushed the mutation. We also increment the space version number.
  • When all the changes have been applied, we invoke the poke request. This is to let replicache know that there are some changes in the database that it should pull. If poke is not done, replicache will eventually pick up the changes when it sends the next pull request. There is no set rule on how to poke the replicache, in our case, we will use supabase realtime.

Implementing poke

+page.svelte

Poke request in an indication to replicache that the database has updated or new records and it should immediately pull the changes. Here is how we do it for our app.

  • Turn on the realtime on the todos table in supabase
  • Listen to changes in the app, preferably +layout.svelte and call replicache.pull({ now: true }); when the change is notified. That’s it

Implementing pull endpoint

/api/pull/+server.ts

In push request, we create, update and delete (soft-delete) the records. Against each record, we maintain under which space version the changes were done and we increment the space version.

Now, the pull request sent by replicache has following properties

Here two properties of interest are cookie and clientGroupID . Cookie is the last space version that was synced and clientGroupID is the client group that wants to sync the new changes if any. Here are the steps followed to reply with the changed records.

  • Get the last space version that was synced. requestCookie here.
  • Get all the records that were changed for the requesting spaceID (user) since last sync requestCookie.
  • Get the last mutation ids of all the clients in the requesting client group. We send the lastMutationIDs as our response to the pull request to the client. This tells the client that these particular mutations were applied and now it should stop sending push requests for these particular mutations.
  • Get the current version of the space. We will send this in the response to the pull request to tell the client that until which space version the changes have been synced. This space version becomes the requestCookie in the next pull request.
  • Lastly, we prepare the response. The response has this shape:

Here path is an array of objects containing multiple operations depending on the status of the record. The shape of each path object is

5.1. Put operation: For all the new records, the operation is put.

5.2. Del operation: Remember, it is important with replicache that we do not hard delete any record. Instead, we set “deleted”: true, when we want to delete a record. By doing, this we can send this record with del operation to the client and replicache will know to delete these records from the local copy, and it will no longer be displayed in the UI.

At this point, if you go to the developer tools of your browser and go to the Application tab, under indexed db, you should see something like this. Make sure you initialised replicache.

UI Implementation

UI is simple. I will explain the relevant parts here.

Mutators

mutators.ts

Mutators are the javascript functions that create, update or delete records on the local and then the same mutations are replayed on the server via push endpoint.

An example createTodo mutator

Here each mutator receives 2 params, a WriteTransaction and the arguments you pass to the mutator. Same arguments are then sent to the server in push request. Here tx.set(key, value) accepts a key and a value. Key here should be the same as the ones you returned in the PatchOperations in the pull endpoint response.

How to call the mutators on button click

+page.svelte

One important thing to note here is the id. The responsibility to generate ids of your records lies with the client. Because the updates are applies optimistically to the local first and then they are sent to the server.

Query subscription

+page.svelte

You can subscribe to any query with replicache to receive changes live as they happen and to update the UI in response.

The changes are received in the onData callback and you should update the UI from here. Don’t forget to unsubscribe when the component is destroyed.

Conclusion

Getting hang of the entire setup takes some time. But I recommend reading and re-reading this and once you implement this yourself, you will start to understand the concepts.

Also, the strategy explained here is known as “Per space version strategy”. There are 2 more, and you can read about them here. Select the one that makes most sense to you and your use case.

If you want to implement replicache in your existing application, I recommend adopting it incrementally. That is port one part of your app and test it before porting the others.

You can see replicache implemented in production on Shootmail.

If you have any questions, hit me up in the twitter DMs. Here is my profile.

Happy coding : )