Because the client requests pagination by lastEventId (a UUID), the server needs to remember every event forever in order to correctly catch up clients.
If instead the client paginated by lastEventTimestamp, then a server that for any reason no longer had a particular event UUID could at least start at the following one.
tinodb 6 hours ago [-]
That’s why the article suggests using a uuid v6 which is time orderable. Or prefixing with an incrementing db id. So indeed, if you intend to delete events, you might want to make sure you have orderable ids of some sort.
hombre_fatal 14 hours ago [-]
It could use a section on high level justification / inspiration.
For example, what inspired this over a typical paginated API that lets you sort old to new with an afterId parameter?
Never heard of "CloudEvents" before. How do people feel about those?
junto 3 hours ago [-]
Here’s a nice comparison between CloudEvents and AsyncAPI from 2019. You can combine them. In the end being able to version and wrap events is useful, although amusingly it reminds me of SOAP.
I enjoy anything that drives down NIH, or something that an existing library could possibly support, or something that I could take to my next job (or could possibly hire for)
I believe cloud events are most common in Kafka-adjacent or event-driven architectures but I think they're used in some GCP serverless things, too
paulddraper 7 hours ago [-]
Good
lud_lite 14 hours ago [-]
What happens if you need to catch up? You keep calling in a loop with a new lastEventId?
What is the intention there though. Is this for social media type feeds, or is this meant for synchronising data (at the extreme for DB replication for example!).
What if anything is expected of the producer in terms of how long to store events?
dgoldstein0 10 hours ago [-]
Sounds like it. But the compaction section has more details - basically you can discard events that are overwritten by later ones
fefe23 13 hours ago [-]
This is an astonishingly bad idea. Don't do this.
Use HTTP server-sent events instead. Those can keep the connection open so you don't have to poll to get real-time updates and they will also let you resume from the last entry you saw previously.
Yeah, but in real life, SSE error events are not robust, so you still have to do manual heartbeat messages and tear down and reestablish the connection when the user changes networks, etc. In the end, long-polling with batched events is not actually all that different from SSE with ping/pong heartbeats, and with long-polling, you get the benefit of normal load balancing and other standard HTTP things
xyzzy_plugh 12 hours ago [-]
Correct. In the end, mechanically, nothing beats long polling. Everything ends up converging at which point you may as well just long poll.
mikojan 7 hours ago [-]
But SSE is a standard HTTP thing. Why would you not be able to do "normal load balancing"?
I would also rather not have a handful of long-polling loops pollute the network tab.
jpc0 4 hours ago [-]
“Normal load balancing” means “Request A goes to server A”, “Request B goes to server B” and there is no state held in the server, if there is a session its stored in a KV store or database which persists.
With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Some proxies / load balancers don’t like long lasting connections and will tear them down if there has been no traffic so your need to constantly send a heart beat.
I have deployed SSE, I love the technology, I wouldn’t deploy it if I don’t control the end devices and everything in between, I would just do long polling.
kiitos 2 hours ago [-]
Your description of "normal load balancing" is certainly one way to do load balancing, but in no way is it the presumptive default. Keeping session data in a shared source of truth like a KV store or DB, and expecting (stateless) application servers to do all their session stuff thru that single source of truth, is a fine approach for some use cases, but certainly not a general-purpose solution.
> With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers.
Weird take. SSE is inherently stateful, sure, in the sense that it generally expects there to be a single long-lived connection between the client and the server, thru which events are emitted. Purpose of that being that it's a more efficient way to stream data from server to client -- for specific use cases -- than having the client long-poll on an endpoint.
Alifatisk 3 hours ago [-]
Isn't SSE limited to like 12 tabs or something? I remember vividly reading about a huge limitation on that hard limit.
curzondax 2 hours ago [-]
6 tabs is the limit on SSE. In my opinion Server Sent Events as a concept is therefore not usable in real world scenarios as of this limitation or error-detection around that limitation. Just use Websockets instead.
If instead the client paginated by lastEventTimestamp, then a server that for any reason no longer had a particular event UUID could at least start at the following one.
For example, what inspired this over a typical paginated API that lets you sort old to new with an afterId parameter?
https://www.asyncapi.com/blog/asyncapi-cloud-events
I believe cloud events are most common in Kafka-adjacent or event-driven architectures but I think they're used in some GCP serverless things, too
What is the intention there though. Is this for social media type feeds, or is this meant for synchronising data (at the extreme for DB replication for example!).
What if anything is expected of the producer in terms of how long to store events?
Use HTTP server-sent events instead. Those can keep the connection open so you don't have to poll to get real-time updates and they will also let you resume from the last entry you saw previously.
https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...
I would also rather not have a handful of long-polling loops pollute the network tab.
With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Some proxies / load balancers don’t like long lasting connections and will tear them down if there has been no traffic so your need to constantly send a heart beat.
I have deployed SSE, I love the technology, I wouldn’t deploy it if I don’t control the end devices and everything in between, I would just do long polling.
> With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers.
Weird take. SSE is inherently stateful, sure, in the sense that it generally expects there to be a single long-lived connection between the client and the server, thru which events are emitted. Purpose of that being that it's a more efficient way to stream data from server to client -- for specific use cases -- than having the client long-poll on an endpoint.
(Details in the previous thread on HTTP Feeds: https://news.ycombinator.com/item?id=30908492 )