Radically improve the performance of Backend Workflows on Bubble.io (+474% faster)

You've probably already noticed:

Compared to Xano or Airtable, Bubble is slow in database operations.

At first, I would say yes.
Then after hundreds of hours of pushing the limits of Bubble with Victor at Flusk, I would say not so much.

At least not as slow as you all think.
Let me prove it to you. 🚀

Prelude

Before we get into the nitty-gritty, you'll need to know a few things.
Here is what you need to know before you start:

• Using backend workflows
• Making recursive backend workflows

‍
For those who don't know, let me give you a concrete example of use:

📚 You want to send an email to each user of your app. You have two solutions:

1. Manually send an email to each user
2. Loop over all your users, and use an action like "Send Email" on each loop round.
   

‍
This is the perfect case where we use recursive backend workflows.

If you want to learn more, I invite you to check out Matt's magic tutorial.

The problem with Recursive Backend Workflows (RBW)

Let's say we want to create or modify a large amount of data in a database. Let's say, for example, that we want to create 8000 books with any name in our "Book" database (which I just created as an example).
So far, no problems on the surface.
Doing a BWR on such a large number of iterations is going to be (very) long, and we probably won't have created all our books before the next hour.

Measures

To measure how long it will take Bubble to create 1000 rows in the database, I will introduce a new Data Type that I call "Summary".

If you look at the fields it contains, you should quickly understand its interest. 
It is a kind of summary with statistics on the 1000 items I just created.
I create a new "Summary" when I start creating my 1000 entries, and I save the exact date I started creating.
Then, when I get to my 1000th item, I save the exact date again.

This allows me to do a simple subtraction to find the number of seconds to create 1000 objects in the database.

So how long does it take to create exactly 1000 items in the database?

For 1000 lines, with a simple RBW on an Agency plan, Bubble takes 402 seconds (6 minutes 42 seconds) to create 1000 lines.
That is 2.49 items / second.
Compared to Xano for example, this is really slow.

But let's see together how we can optimize this!

The solution

You didn't come to this article to learn what you already know.
I'm going to teach you how to make this BWR up to +200% faster (at first).
And that's with a simple solution.

‍
Here's what we do now for each loop:

• We create a "Book".
• We restart our BWR by incrementing the index, provided that the current index is not higher than the limit.

At Flusk, we've noticed a small subtlety that has huge impacts on BWR performance.
What if instead of creating only one "Book" per loop turn, we created several?
For example, what if we created 3 books per loop?

Here are the results:

We managed to create 1000 items in the database in (only) 192 seconds (3 minutes 12 seconds), that is 5.2 items / second.
Almost 2x faster than the previous method!
More precisely, our operation is +109% faster compared to the beginning.

‍
And here I see you coming.

"Why don't we put 5, 10, 15 actions per loop turn?"

And you're absolutely right, that's what we'll do. Maybe not 15 actions, but we can try with 5 and 10.

Small improvement

In order to make it more maintainable, we will group our 3 creation actions in a custom workflow that we will call "create-book". Here is what it looks like:

And now we will call this custom workflow 3 times (instead of directly creating 3 "Books"). So our new workflow looks like this:

The main reason to create a custom workflow is that depending on how you use this technique, you may want to do several actions instead of one. For example, you may want to create a company and then add it to your Current User.
This is equivalent to two actions, so you would have to create 6 steps in your workflow.

By creating a custom workflow that does these two actions, your workflow will only have 3 steps (the 3 triggers of the custom workflow, which has 2 steps. 2x3 = 6, so we have our 6 steps).

5 actions per loop

Here are the results for 5 creations per loop:

We are still improving!
133 seconds (2 minutes 30 seconds), and 7.52 items / second (+202% compared to the start)

10 actions per loop

Let's see now with 10 creations per loop :

Even better!
In 105 seconds (1 minute 35 seconds), with 9.52 items / second (+283%!).

I can see the burning question coming:
What is the limit? How far can we go?

35 actions per loop

We'll jump right to 35 and see what happens.
Here's what my BWR looks like now (I'm ashamed 😅):

And now let's look at the result:

Yes, it gets really fast!
But you've seen my workflow... nothing maintainable about it.

‍
Anyway, here are the statistics we came up with:
1000 items created in 70 seconds (1 minute 10 seconds), that's 14.29 items / second (+474% from the beginning).

⚠️ Warning

What you might not think about is the capacity of the Bubble server.
What we've done here isn't magic either.

We could take the example of a car on the highway:
If you drive at 90 km/h, you don't consume much but you will arrive at your destination later.
On the other hand, if you drive at 200 km/h, you will consume a lot more but you will arrive at your destination much faster.
With Bubble, it's the same.

You're asking for a lot more resources from your server at once, and you have to be careful about that.
When you increase the number of operations per loop, you increase the power consumption of your server (the processor works more, like an engine running faster)

⚠️ But watch out for the heart attack!
You can monitor all this with the CPU consumption graphs provided by Bubble, in the "Logs" > "Capacity" tab.
It all depends on the monthly subscription you have chosen on your application.

⛔️ Furthermore, it is clearly not a good practice to operate like this.
It is not at all maintainable to have +30 identical actions in the same workflow. Even if using custom workflows helps us if we ever need to make a change, it's still not a good practice and that's probably why Bubble doesn't communicate about it.
This tip can be useful to know, and to apply in some very specific and single use cases.

Example:
You want to add to your 10,000 users a relational data "Company", or something like that.
This is something that happens only once, and therefore by definition will not be maintained.

Conclusion

So here is what we can learn from this experiment:

It is indeed (much) more efficient to place multiple actions in each loop round rather than just one.
We "stack" the operations, and I have the impression that this is where Bubble loses time, it is in the re-scheduling of the next iteration.

Here are the final results:

Anyway, I hope this will be useful to you.
Just keep in mind that this is not a "clean" way, and it will never be taught to you in any no-code training. And there's a reason.
If you like articles like this, feel free to give me feedback.
I'd also like to see who will go the distance and put +100 operations per loop turn 🤣 (mention me!)

Here is the link to the Bubble editor.
In the meantime, you can:

• Go check out what we're doing at Flusk 🚀
• Connect with me on LinkedIn 🕺
• Simply contact me via email (wesley@flusk.eu) if you have any questions/requests in relation to this article!

See you,
Wesley

‍

‍

‍

‍

‍

‍