Pointing http resources to dev servers in Retool with mitmproxy

Thanks to Devang for suggesting mitmproxy and his ideas towards this approach!

If you use retool and have found yourself having to change the resource underlying a number of queries for local development, this could be a helpful addition to your workflow. Consider a retool app that has many queries that access a similar http resource that your team has built:

Suppose this app needs an update and you'd like to develop a new api on your local system. You decide to setup a releases workflow and go in and change all the resources to your local system's address (with something like tailscale, zerotier, or ngrok). Once done with your changes, you have to manually change everything back to the main resource after deploying. That's quite a pain and could cause outages where 9/10 queries have been changed back but not the 10th query.

We'd like a safe and quick way to be able to point traffic for local development without having to always remember to change the http resource in the primary app back and forth. One approach to doing so is by introducing a mitmproxy between retool and your primary resource. You'll first need a way to tell retool who is accessing the resource (globally across all apps) - by setting the following header X-Retool-User:

That's all you need to do from retool standpoint. Now here's where the real magic comes in with mitmproxy. The current architecture is:

We now will make it:

All you need is redis which includes keys of the user with the proper host to point their traffic towards (you could even have one team member going to another team members machine). Here's a quick snippet of this rule in mitmproxy which runs as a python reverse proxy process:

You can do a lot more advanced things, like have the redis be get/set/list from the mitmproxy code itself (the above screenshot is simplified but you could imagine doing it above), doing strong access control, replacement and scrubbing of access tokens, etc.

MITMProxy is quite powerful and isn't limited to just retool apps but could be extended to many more things even outside reverse proxy mode (other reverse proxies include nginx / openresty / kong / aws api gateway - where you could do the same thing). You can use this trick with any http resource (even those you don't own), however you cannot do this with databases or retool built in resources (since it's hard in those cases to set a global header indicating who the requester is).