Low file descriptors limit (1024) with AMI or OVF images causes errors

User Impact: Prometheus can fail with "Too many open files" error on AWS and OVF PMM2 Server
STR:

1. Run AWS or OVF PMM2 server

2. Check limits

Given result: Max open files 1024 4096 files
Expected result The limit should be increased.

Original report:
PMM instances deployed in AWS Marketplace, have very incompetent limits set for open file descriptors (1024) by default. This can fail prometheus  with Too many open files issues and can happen in many different work flows.

I have created two PMM instances one in AWS Market place and the other in docker containers. You can see the file descriptors limit set in both the environments and there is huge difference between them.

AWS Marketplace:

Docker container:

  Docker container has 1048576 where as AWS instance has 1024 which is far less than its counterpart.

Workaround: 

One needs to create an override file using systemctl edit as given below.

 Sample Error:

This limit needs to be as high as in docker container variant.