Node.js, redis, and resque —
Interesting use of node.js as a sort of HTTP reverse proxy. It uses redis based queues to communicate with backends instead of establishing a direct socket connection and doing HTTP:
This spike uses node to put messages into a (redis) queue. Ruby background workers read from the queue, process the requests, and respond on a different queue. When node receives the response from the background worker, it sends the response back to the waiting user.
I assume this adds a not insignificant amount of latency to each request but would also make possible a bunch of long-running connection features. For example, the response (or portions of the response) could be delivered from separate worker processes. This style of architecture, where the client connection isn’t tied to backend web process, looks promising. The nginx_http_push_module is another example that gives the same types of benefits.
pgrs.net
14:26
Resque —
Really excited to see Chris release the shiny new Redis-based work queue that’s been running GitHub for the past couple of months:
It boils down to this: GitHub is a warzone. We are constantly overloaded and rely very, very heavily on our queue. If it’s backed up, we need to know why. We need to know if we can fix it. We need workers to not get stuck and we need to know when they are stuck. We need to see what the queue is doing. We need to see what jobs have failed. We need stats: how long are workers living, how many jobs are they processing, how many jobs have been processed total, how many errors have there been, are errors being repeated, did a deploy introduce a new one?
I’m still getting my feet wet with the jobs system but I can’t wait to get my hands dirty in the guts of this thing.
github.com
04:26