I am setting up infrastructure for a startup which pretty much will not have a lot of traffic, but should be able to scale when needed.
We are favouring a setup with a LB, that distributes traffic to the frontend nodes in a dedicated private subnet (over 3 availability zones), which in turn make request to the backed nodes on their own dedicated private subnet, which in turn makes requests to mongodb managed via atlas and vpc peering.
In order for each node to provision it requires internet access. The backend nodes also make request to third party services and therefor require internet access when they are running as well.
I see three options:
set up a nat gateway for each private subnet in each availability zone. Depending on location this amounts to around 30$ per subnet per availability zone. With 3 availability zones and 2 subnets this will total to around 180$ a month, which is actually more than we plan to use for the ec2 instances while there is not much traffic and load on the system. We could probably cut that down to just use 1 nat gateways in each availability zone for all the private subnets, but still that's around 90$.
set up ec2 instances as nat gateway, which will probably be a little cheaper, however requires maintenance and setup.
just use one private subnet, assign public ips to each node and use the internet gateway, via route table entries. I don't think using dedicated private subnets will make much sense as the nodes should be able to connect with each other via the gateway anyway.
The last option will most likely be the cheapest option as one elastic ip is already included within an ec2 instance and dedicated gateways are not needed. However I was wondering if there is a significant downside or risk involved in doing so? We plan to return to the idea with dedicated subnets when there is a need to (like there is significant traffic), but we really would like to keep costs as a low as possible in the beginning.