Storing logs to one server and displaying data – Affiliate Marketer Job

I have 50 servers, which are emitting logs via pm2 (to .log files) approximately every 30 seconds. The logs are all in text form and will need to be manipulated to gather the relevant data, and then display this data on graphs.

For the graphs, the X axis will be time, and should be adjustable (i.e. I should be able to narrow or expand the timeframe I’m looking at). Since the logging is provided every 30 seconds, and I will likely be looking at the data on a daily, weekly, or even 2-3 month basis, the ability to aggregate / reduce the number of points on the graph will be important to ensure the graph is not slowed down / laggy.
The Y axis should be one of 5 variables (and I should be able to look at 1,2,3,4, or all 5 variables at once). Each of these 5 variables should be plotted on their own Y axes, so they all fit on my screen. As I hover across the graph, I should be able to see the precise values for the variables that I have selected to be displayed. .

The solution should be able to handle up to 200 servers of logging.

In my mind, there are two pieces to this solution. One piece is something that will run each of the 50 servers. The other is something that runs on a new server that aggregates all the logs and handles the graphing, or distributes the data for the graphs.

In your proposal, please let me know which languages you plan on using, the core technologies (especially which graphing technology you’ll use), and the high-level structure of your solution (i.e. will you be using cron, or reading the log file every time, or streaming the log file).

I would prefer this work done in python or NodeJS, but am open to other languages (JS or not). I’ve historically used Grafana for graphing but am unsure if it’s appropriate here.

Budget: $500

Posted On: April 11, 2024 19:22 UTC
Category: Full Stack Development

Country: United States

click to apply