- Author: Nikita Prokopov
- Full Title: Local, First, Forever
- Category: articles
- Document Tags: #tech
- URL: https://tonsky.me/blog/crdt-filesync/?utm_source=hackernewsletter&utm_medium=email&utm_term=fav
Highlights
- Here’s my short version:
• It’s software.
• That prefers keeping your data local.
• But it still goes to the internet occasionally to sync with other users, fetch data, back up, etc. (View Highlight)
- But file syncing is a “dumb” protocol. You can’t “hook” into sync events, or update notifications, or conflict resolution. There isn’t much API; you just save files and they get synced. In case of conflict, best case, you get two files. Worst — you get only one :) (View Highlight)
- This simplicity has an upside and a downside. The upside is: if you can work with that, you would work everywhere. That’s the interoperability part from Martin’s talk. (View Highlight)
- Version 1: Super-naive (View Highlight)
- Let’s just save our state in a file and let Dropbox sync it (in my case, I’m using Syncthing, but it’s the same idea. From now on, I’ll use “Dropbox” as a common noun). (View Highlight)
- CRDT is a collection of data types that all share a very nice property: they can always be merged. It’s not always the perfect merge, and not everything can be made into a CRDT, but IF you can put your data into a CRDT, you can be sure: all merges will go without conflicts. (View Highlight)
- Version 2: A file per client (View Highlight)
- The only way to avoid conflicts is to always edit locally. So let’s give each client its own file! (View Highlight)
- Now we just watch when files from other clients get changed and merge them with our own. (View Highlight)
- Version 3: Operations-based (View Highlight)
- Note: will require some research
- What if your CRDT is operation-based? Meaning, it’s easier to send operations around, not the whole state? (View Highlight)
- You can always write operations into a separate append-only file. Again, each client only writes to its own, so no conflicts on the Dropbox level: (View Highlight)
- Now, the operations log can grow quite long, and we can’t count on Dropbox to reliably and efficiently sync only parts of the file that were updated. (View Highlight)
- In that case, we split operations into chunks. Less work for Dropbox to sync and less for us to catch up: (View Highlight)
- You can, of course, save the position in the file to only apply operations you haven’t seen. Basic stuff. (View Highlight)