A few weeks back, wanted to take a break from the downhill rush and get back to building things. As is almost always the case, took to the trusted crisp pipeline and building something on top. One of the reasons is the fact that muscle memory with a lot of things makes it a low-hanging fruit. More so, when it comes to building things when the focus is personal validation to break cycles.
The context thus set, was looking at working three limbs of the Octopus* out, to make them slim yet robust. First, shedding weight on the pipeline process that fetches our news content. Second, now that Node has good enough support (tbh, has been for long - SMH), engage WorkerTheads. Third, and not the least, introduce SSEs into the system.
Since the first two aren't all that new to the system or otherwise, let me switch gears to the third. So, what are SSEs (Server-Sent Events)? In one birth/realm of mine, I would call them the holy grail but that's too narrow after all.
As developers, we all have employed long-polling to get the updated state from servers. What the client doesn't know in this case, is how often to place the requests. And with lower bandwidth/older hardware, this is much avoided the better.
So, why not something like Websockets? Of course, a good choice since it is bi-directional - both the client and server can communicate. But imo, the two tend to solve two different use cases as far as implementations go. For cases like say, a chat room, SSEs don't make much sense, given the bi-directional nature of data transfer. But, for a resource dashboard, all data pushed out from a data source makes SSEs a better choice.
With this out of the way, added a data API my HTTP server + SSEs. And the beauty of it all was that a bare-bones server didn't have to do much to support SSEs.
To cut straight to the chase, the following are all that's needed.
Server writes the
text/event-streamheader once when it first responds. This keeps the connection open for further responses.All response must get padded with
data:and '\n\n' - think of this as communication over a walkie talkie/RF receiver. This helps the client identify where the message ends.In a periodic manner, the server pushes the response to the client as above.
The client, initialises an EventSource to the server and listens on it. All data gets passed via the onmessage event's data property for the client's use.
Optimising this further, as always, is a question of what and where and when. For a starter, one could diff the data entries and construct a graph of operations to the client. Or, return the entire payload for a client-side entity like React to diff things.
One take-away from experience is the push to take the right tipping point for adoption. SSEs, have >96% support in the wild and has been there for a while. I've experimented with Tornadoweb's periodic callback, combining it with MongoDB's oplog/changestreams. One of it was a case of a dash for our MongoDB instances and the other, for a stream API. SSE adoption does in a sense provide a few more pointers to how things can be better structured in such cases.
Made a dead-simple repo to get started with. Not much there of course, because there isn't much for the dev to do; focus on your business logic is a good thing after all.