Storing counts in firestore
This is really unbelievable that Firebase Firestore doesn’t have kind of db.collection.count()… But it’s not a reason to drop firebase.
You currently have 3 options:
Option 1: Client-side
Select all from the collection and count on the client-side. This works well enough for small datasets but obviously doesn’t work if the dataset is larger. Handling this on the front end should be fine as long as you are not doing too much logic with this returned array.
db.collection(‘…’).get().then(snap => snap.size) — will return the collection size
Option 2: Write-time best-effort
Medium collection (100 to 1000 documents)
Use with care — Firestore read invocations may cost a lot
This works well for any dataset size, as long as additions/deletions only occur at the rate less than or equal to 1 per second. This gives you a single document to read to give you the most current count immediately.
Handling this on the front end is not feasible as it has too much potential to slow down the user’s system. We should handle this logic server-side and only return the size.
The drawback to this method is you are still invoking firestore reads (equal to the size of your collection), which in the long run may end up costing you more than expected.
Cloud Function:
db.collection(‘…’).get().then(snap => res.status(200).send({length: snap.size}));
Front End:
yourHttpClient.post(yourCloudFunctionUrl).toPromise().then(snap => snap.length ) — will return the collection size
For solution 2, Firebase website has these examples about cloud function that solve a similar problem:
- Distributed counters
- Aggregation queries
Solution: Distributed counters
To support more frequent counter updates, create a distributed counter. Each counter is a document with a subcollection of “shards,” and the value of the counter is the sum of the value of the shards.
Write throughput increases linearly with the number of shards, so a distributed counter with 10 shards can handle 10x as many writes as a traditional counter.
The following code initializes a distributed counter:
To increment the counter, choose a random shard and increment the count:
To get the total count, query for all shards and sum their count fields:
Limitations
The solution shown above is a scalable way to create shared counters in Cloud Firestore, but you should be aware of the following limitations:
- Shard count — The number of shards controls the performance of the distributed counter. With too few shards, some transactions may have to retry before succeeding, which will slow writes. With too many shards, reads become slower and more expensive.
- Cost — The cost of reading a counter value increases linearly with the number of shards, because the entire shards subcollection must be loaded.
Aggregation queries
Advanced queries in Cloud Firestore allow you to quickly find documents in large collections. If you want to gain insight into properties of the collection as a whole, you will need aggregation over a collection.
Cloud Firestore does not support native aggregation queries. However, you can use client-side transactions or Cloud Functions to easily maintain aggregate information about your data.
Solution: Client-side transactions
Consider a local recommendations app that helps users find great restaurants. The following query retrieves all the ratings for a given restaurant:
Rather than fetching all ratings and then computing aggregate information, we can store this information on the restaurant document itself:
In order to keep these aggregations consistent, they must be updated each time a new rating is added to the subcollection. One way to achieve consistency is to perform the add and the update in a single transaction:
Using a transaction keeps your aggregate data consistent with the underlying collection. To read more about transactions in Cloud Firestore, see Transactions and Batched Writes.
Limitations
- Security — Client-side transactions require giving clients permission to update the aggregate data in your database. While you can reduce the risks of this approach by writing advanced security rules, this may not be appropriate in all situations.
- Offline support — Client-side transactions will fail when the user’s device is offline, which means you need to handle this case in your app and retry at the appropriate time.
- Performance — If your transaction contains multiple read, write, and update operations, it could require multiple requests to the Cloud Firestore backend. On a mobile device, this could take significant time.
If client-side transactions are not suitable for your application, you can use a Cloud Function
Most scalable solution
FieldValue.increment()
This ensures we have correct counter values even when updating from multiple sources simultaneously (previously solved using transactions), while also reducing the number of database reads we perform.
By listening to any document deletes or creates we can add to or remove from a count field that is sitting in the database.
Increment is a special value that will atomically update a counter based on the interval you specify. You do not need to read the document to obtain the current total — Firebase does all the calculation serverside. In most cases, you will be keeping a count by adding an integer of 1 to the current total.
Increase a Counter
Decrease a Counter. We can decrement a counter by simply changing the interval value to -1.
Counting Values in Separate Documents
In certain cases, your counter value may depend on the creation of a new document or just be located in a different document. You can use batched writes to ensure all data is updated safely. For example, we might want to create a like document, then update the total multiple places.
Keep a Count of Documents in a Collection
Create a document called — stats — for the purpose of maintaining aggregation of metadata about this collection. In your frontend code, use a batch to increment the stats when creating a new document.
Referencies:
- Firebase. 2021. Aggregation Queries | Firebase. [online] Available at: <https://firebase.google.com/docs/firestore/solutions/aggregation> [Accessed 2 January 2021].
- Firebase. 2021. Distributed Counters | Firebase. [online] Available at: <https://firebase.google.com/docs/firestore/solutions/counters> [Accessed 2 January 2021].
- Delaney, J., 2021. How To Use Firestore Increment. [online] Fireship.io. Available at: <https://fireship.io/snippets/firestore-increment-tips/> [Accessed 2 January 2021].
