Sneak peek: Biff, a Clojurey alternative to Firebase

This week in Findka: I’ve just published A recommender system in 30 lines of Clojure, which explains in detail how Findka’s recommendation algorithm works (it’s really simple).

Besides that, I’m close to releasing the next step in my work with web application architecture: Biff. It can be explained in two different ways:

First explanation: Biff is an open-source, self-hosted, Clojurey alternative to Firebase

I’ve enjoyed working more with Firebase for the past few months, seeing how far I could push it. For my small startup and the other hobby-size projects I work on, I’ve found it to be a big improvement over using AWS + Datomic. However, I’m unlikely to use it for a backend going forward. Firebase has its own limitations:

1. JVM + Clojure has better libraries than Node + ClojureScript.

2. Firestore (and its rules system), while a lot nicer than I expected it to be, is still not as nice as what could be achieved with Datomic or Crux.

3. I’d rather have a long-running JVM process for my backend than ephemeral JS functions. Firebase’s Cloud Functions have to finish running before they return a response, so you can’t return a response and then finish handling a request in the background. Plus, my traffic is low enough that requests often result in cold starts. As a result, I often get 3-5 second response times for certain user actions. Maybe it doesn’t matter at my scale—but I find it annoying.
   
   With a long-running JVM backend, besides not having cold-start issues, you can also use web sockets. Ephemeral functions ostensibly make it easy to scale up and down, but I think scaling JVM instances will work well enough, if not much better.

So, I’m moving to DigitalOcean. Biff will be a managed deployment solution + lightweight, full-stack web framework that includes:

• One-click installation to DigitalOcean, including automatic config for Nginx, Letsencrypt and a firewall.

• Crux as the database (using either the filesystem or DigitalOcean’s managed Postgres for durable storage).

• A rules system based on Spec for authorizing CRUD operations.

• Subscribable queries for Crux, similar to Firebase’s subscribable queries. These won’t be Datalog, but you can join them declaratively on the frontend like I demonstrated with Mystery Cows. I’ll probably (eventually) add complex subscribable queries with Materialize.io.

• A web server (Immutant) + web sockets (Sente) out of the box. Web socket events are dispatched for you on the frontend and backend, so you just have to register multimethods to handle them.

• Session-based authentication. Just passwords at first, but later I’d like to add passwordless email links if you provide SMTP credentials. And also SSO through Google etc.

• Static file serving through Nginx. (Even if your backend goes down, Nginx will keep serving your frontend files). Later I’ll add CDN support.

As time goes on I’ll likely add lots more stuff—like self-hosted analytics.

The whole stack will be open-source and easily hackable. Although it’s a framework, I’m working very hard to make sure it doesn’t get in your way. And also: this might seem like a lot, but it’s really not all that complex. I already have a good chunk of it working. Most of the work has been the past 1.5 years I’ve spent experimenting with different web app architectures.

I am convinced that this stack will make it significantly easier to get small startups and hobby projects up and running quickly. I’m planning to move Findka to this stack, and I’ll make sure Biff scales along with it.

Second explanation: Biff is a package manager for self-hosted Clojure web apps.

This is the explanation given in Biff’s repo, linked above. Biff has a simple plugin system, less than 30 lines of code. The entire web framework is written as a collection of plugins, and your main app is also written as a plugin. Biff includes an admin interface that lets you install plugins directly from Github (it adds the plugins as git dependencies to your deps.edn file). This isn’t needed if you’re using Biff simply as a Firebase alternative, since you could just add the dependencies to your deps.edn file yourself.

However.

If you’re writing an open-source app, Biff makes it dead-easy for people to self-host it on DigitalOcean—or even their local machines if they want to just use it as a desktop app. In that case, all you have to do is push your app to Github. Super easy for everyone.

Also, people can install multiple apps to a single Biff instance. Instead of having your data spread across multiple services and hoping that the app developers have written the integrations you want, you can give multiple apps direct access to the same database.

This can greatly simplify some kinds of applications. Instead of using APIs to transfer the same person’s data from service to service, APIs would only be used for transferring data between different people. If you want to write an app that slightly extends an existing app, it’s a piece of cake. Just read and write directly to the database.

It’s a shame that the shift from desktop to web has pretty much relegated open-source software to the role of building blocks, while the final applications have been predominantly closed-source. We got convenience but sacrificed extensibility. With Biff, I think we can get both convenience and extensibility.

If it works, then open-source software might start eating market share away from closed-source web applications. That could have big implications for the industry. My favorite implication: there will be much more demand for open-source development, and it could thus eventually become normal for programmers to make a living off Github Sponsors. Freedom for my people.

A big reason I chose to be a startup founder was so that I could work on things I cared about. If I had the option right now to work full-time on Biff and get a decent salary for it, I would be tempted. Findka would be easy to continue as a side project.

Which explanation is best?

Although I think the second explanation has bigger impact long-term, the first might be easier for people to grasp right away. So I’m thinking of explaining Biff mainly as a Firebase alternative but then releasing various apps and showing people how to self-host them with Biff.

In other words: maybe the second explanation can be replaced with “show, don’t tell.” I have a slew of apps I’d like to port to Biff (and some new ones I’d like to write). I could just say “check out all these apps; just clone Biff and then you can run them all locally. And if you want to run them in the cloud, there’s a one-click DigitalOcean install.”

Solo hacker’s guide to Clojure

In either case for Biff, providing stellar documentation and other learning resources is a top priority for me. I’m planning to use my Clojure guide to teach Biff. After you learn how to make Biff apps, I’ll show how Biff itself is written, so readers will understand how things are working under the hood. Even if you don’t want to use Biff, you’ll have learned how to put together a full Clojure stack on your own.