Relay + Envoy

[dorrogeray]

Hi, I just noticed that Relay is not working with Envoy proxy. This is likely because there is a feature gap in Envoy where it does not support the HELLO command. I just wanted to bring this to your attention, maybe there is a way to make Relay configurable in a way where it does not need to send the HELLO command, thus making it compatible with the Envoy until they can fix this feature gap.

For completeness, the error we get is: unsupported command 'HELLO' (RELAY_ERR_REDIS; commands.c:8104). I understand that this just what relay forwards from Envoy.

[michael-grunder]

Thanks for the question, I'll look into it.

[tillkruss]

@dorrogeray: What version is the Redis Server running behind Envoy?

[dorrogeray]

@tillkruss 6.2.6, but to be clear, HELLO works just fine on the redis, its the Envoy who does not support proxying of HELLO command.

[michael-grunder]

My assumption here is that envoy doesn't support the HELLO command because it doesn't support RESP3.

This presents a problem for Relay because it was designed from the ground up to use version 3 of the Redis protocol (primarily to support push messages for key invalidations).

Envoy also doesn't seem to support the client command, meaning we can't enable server-assisted client-side caching either. In and of itself that's fine because Relay can be configured to not cache data which effectively makes it behave like PhpRedis but the RESP2/RESP3 distinction would require a fair bit of logic on our end to support.

Not impossible but there would need to be a pretty high demand to make such changes worth doing.

[dorrogeray]

Thanks, that makes sense. 👍 After taking a closer look, I realized that you are right and Envoy indeed does not support RESP3 at all, its not just the "HELLO" command. 😞 Any scalable RESP3 proxy you can recommend for connection pooling? 🤔

I suppose that writing a RESP3 compatible proxy is non-trivial if you want to support push messages, since the proxy would have to forward the pushes from redis into the correct client connection (meaning, you would always have to keep clients which are subscribed connected to the same redis connection, you could not move them around at will between the redis connections)..

Maybe there is a different angle to this. Could relay support single persistent redis connection per php-fpm worker pool? 🤔 For example, our php-fpm pools have 40 workers, so we could save up to 40x persistent connections if relay was able to share the connection. Then you would not need proxy like Envoy at all.

[michael-grunder]

Appologies for the delay, I didn't see your final question until now.

Maybe there is a different angle to this. Could relay support single persistent redis connection per php-fpm worker pool? 🤔 For example, our php-fpm pools have 40 workers, so we could save up to 40x persistent connections if relay was able to share the connection. Then you would not need proxy like Envoy at all.

So the short answer here is yes, it is technically possible.

The longer answer is that it requires somewhat exotic mechanisms in the kernel. On newer kernels you can use the pidfd_getfd syscall to share a kernel file descriptor between unrelated processes. This does require giving the worker processes elevated privileges (PTRACE_MODE_ATTACH_REALCREDS ). On older kernels you can use AF_UNIX sockets and SCM_RIGHTS to share fds.

It wouldn't exactly be one fd per php-fpm pool, but N connections shared between each php-fpm worker. That said, you could set N = 1. When the first worker opened a connection, a different worker can later grab ownership of it and use the same underlying fd.

The most complex part in doing something like this is managing the concurrency and connection state. Since the underlying connection is shared between workers, we need to ensure a worker doesn't leave the connection in a MULTI block, for example.

[dorrogeray]

Thanks for the reply! Its great to know that its technically possible. 👍