Introduction

Managing logs efficiently is crucial for understanding the behavior of your application, diagnosing issues, and ensuring smooth operations. After evaluating different solutions based on pricing and feature set, we determined that sending logs to Google Logs Explorer via a custom Node.js application is the most cost-effective approach.

In this article, we will cover:

• Setting up a Node.js application to receive logs from Heroku.
• Configuring Google Cloud Logging to store logs.
• Deploying the application to Heroku.
• Setting up a Heroku log drain to send logs to our application.

Setting Up the Node.js Application

Our application uses Express to receive logs from Heroku and winston with Google Cloud Logging for structured logging.

1. Create a new project

mkdir heroku-log-handlercd heroku-log-handlernpm init -y

2. Install dependencies

npm install express body-parser winston @google-cloud/logging-winston

3. Application Code (index.js)

The following code receives logs via HTTP POST requests, filters out unnecessary logs, and sends them to Google Logs Explorer.

const express = require('express');const bodyParser = require('body-parser');const winston = require('winston');const { LoggingWinston } = require('@google-cloud/logging-winston');const app = express();const port = process.env.PORT || 4000;const projectId = process.env.GOOGLE_CLOUD_PROJECT;const loggingWinston = new LoggingWinston({ projectId });app.use(bodyParser.text({ type: '*/*' }));const logFilter = winston.format((info) => {    const excludeStrings = ['Worker', 'assets', 'post_office', 'packs'];    for (const str of excludeStrings) {        if (info.message.includes(str)) {            return false;        }    }    return info;});const logger = winston.createLogger({    format: winston.format.combine(        logFilter(),        winston.format.timestamp(),        winston.format.json()    ),    transports: [loggingWinston]});app.post('/', (req, res) => {    const message = req.body;    let httpRequest = {};    if (message.includes('method=') && message.includes('status=')) {        const logParts = message.split(' ');        const logDetails = {};        logParts.forEach(part => {            const [key, value] = part.split('=');            logDetails[key] = value;        });        httpRequest = {            status: logDetails.status,            requestUrl: logDetails.path?.replace(/"/g, ''),            requestMethod: logDetails.method,            remoteIp: logDetails.fwd?.split(',')[0]?.replace(/"/g, ''),            serverIp: logDetails.fwd?.split(',')[1]?.replace(/"/g, ''),            protocol: logDetails.protocol        };    }    const logLevel = message.includes('FATAL') ? 'error' :                     message.includes('postgres') ? 'warn' :                     'info';    logger.log({        level: logLevel,        message,        httpRequest    });    res.status(200).send('Logs received');});app.listen(port, () => {    console.log(`Log handler app listening on port ${port}`);});

Configuring Google Cloud Logging

1. Enable Google Cloud Logging

• Go to the Google Cloud Console.
• Navigate to Logging > Logs Explorer.
• Enable the Cloud Logging API.

2. Set Up Authentication

Instead of using a service account file, set GOOGLE_APPLICATION_CREDENTIALS:

heroku config:set GOOGLE_APPLICATION_CREDENTIALS=/app/service-account.json

Upload the JSON file to Heroku (use this only for local development; in production, use IAM roles):

heroku config:set GOOGLE_CLOUD_PROJECT=your-google-cloud-project-id

Deploying to Heroku

1. Create a Procfile

Create a Procfile in the root directory:

web: npm start

2. Initialize a Git Repository

git initgit add .git commit -m "Initial commit"

3. Create a Heroku App

heroku create your-heroku-app-name

4. Deploy to Heroku

git push heroku main

5. Scale Dynos

heroku ps:scale web=1

Configuring Heroku Log Drain

To send Heroku logs to our application:

heroku drains:add https://your-heroku-app-name.herokuapp.com/

To list existing drains:

heroku drains

To remove a drain:

heroku drains:remove https://your-heroku-app-name.herokuapp.com/

Viewing Logs in Google Logs Explorer

1. Open Google Cloud Console.
2. Go to Logging > Logs Explorer.
3. Use filters to search for your logs.
4. Verify that logs are correctly stored.

Conclusion

After evaluating various solutions, we found that sending logs to Google Logs Explorer via this approach is the most cost-effective solution. Other options such as third-party logging services often introduce pricing concerns due to data retention policies and ingestion limits.

By leveraging Heroku log drains and Google Cloud Logging, we ensure scalable and structured logging with minimal costs.