Sending Email alerts with Log4j – Controlled Alerts

In one of my recent posts I wrote about sending email alerts using log4j. The SMTPAppender supplied with log4j is nice, but it a lacks of a basic important feature, when it comes to sending mail alerts: mails sent are not controlled. On every exception logged by log4j a mail message is sent. This is not always a good thing, since there are times servers may encounter massive amount of exceptions in a short period of time. When such a thing happens we don’t always care about all the exceptions, usually because when big amount of exceptions happen on a short period of time, there is a high probability they are all of the same type and happen from the same cause.

For example, the server may be trying to send some information over the network to a remote machine. If the server fails, it tries to send the information again after one second. Suppose the network is going down for 30 minutes. This will cause the server to send big amount of email alerts. When actually it is necessary to receive only one email alert notifying about the problem.

For this reason, I wrote an email alerts appender that extends the capabilities of the SMTPAppender supplied by log4j, to add basic level of controllability over the mail alerts send by log4j. The appender simply gives adds 2 new parameters:

• TimeFrame: Time frame in minutes.
• MaxEmails: Maximum allowed email alerts.

Both parameters restricts the maximum allowed email alerts that can be send in a given time frame.

For example, you can define that the maximum amount of emails in 30 minutes is 10. In a case of a severe server problem that generates big amount of email alerts, the appender simply stops sending mails after 10 messages were already sent. After 30 minutes, the server will allow mails again. This is not an ideal solution, but it is simple and neat, and it answers most of the needs from a basic alerts mechanism.

The appender is constructed from 2 classes:

• BaseFilteredSMTPAppender: which is an abstract class that defines the basic behavior of the controlled smtp appender.
• FilteredSMTPAppender: which extend BaseFilteredSMTPAppender to create the actual controlled filtering of email alerts.

This is the code of the BaseFilteredSMTPAppender:

import org.apache.log4j.net.SMTPAppender;
import java.util.Date;
import java.util.List;
import java.util.ArrayList;
import java.util.Iterator;
public abstract class BaseFilteredSMTPAppender extends SMTPAppender {
  private int timeFrame;
  private int maxEMails;
  protected long timeFrameMillis;
  protected List<Date> exceptionDates = new ArrayList<Date>();
  public int getTimeFrame() {
    return timeFrame;
  }
  public void setTimeFrame(int timeFrame) {
    this.timeFrame = timeFrame;
  }
  public int getMaxEMails() {
    return maxEMails;
  }
  public void setMaxEMails(int maxEMails) {
    this.maxEMails = maxEMails;
  }
  @Override
  public void activateOptions()
  {
    super.activateOptions();
    timeFrameMillis = timeFrame * 60 * 1000;
  }
  protected void cleanTimedoutExceptions()
  {
    Date current = new Date();
    // Remove timedout exceptions
    Iterator<Date> itr = exceptionDates.iterator();
    while (itr.hasNext())
    {
      Date exceptionDate = itr.next();
      if (current.getTime() - exceptionDate.getTime() > timeFrameMillis)
      {
        itr.remove();
      }
      else
      {
        break;
      }
    }
  }
  protected void addException()
  {
    exceptionDates.add(new Date());
  }
  protected boolean isSendMailAllowed()
  {
    return exceptionDates.size() < maxEMails;
  }
}

And the code of the FilteredSMTPAppender which is much simpler and contains only the actual filtering:

import org.apache.log4j.net.SMTPAppender;
public class FilteredSMTPAppender extends BaseFilteredSMTPAppender {
  @Override
  protected void sendBuffer()
  {
    cleanTimedoutExceptions();
    if (isSendMailAllowed())
    {
      super.sendBuffer();
      addException();
    }
  }
}

And log4j configuration is very similar to the one already posted on the previous blog dealing with email alerts, besides adding the 2 new parameters: TimeFrame, MaxEmails. Here is an example if how it looks:

log4j.rootLogger=INFO, a, email
log4j.appender.a=org.apache.log4j.ConsoleAppender
log4j.appender.a.layout=org.apache.log4j.PatternLayout
log4j.appender.a.layout.ConversionPattern=%d{HH:mm:ss} %-5p [%c{1}]: %m%n

log4j.appender.email=com.bashan.log4j.FilteredSMTPAppender
log4j.appender.email.BufferSize=10
log4j.appender.email.SMTPHost=mysmtp.mailserver.net
log4j.appender.email.SMTPUsername=myusername@mycompany.com
log4j.appender.email.SMTPPassword=mypassword
log4j.appender.email.TimeFrame=30
log4j.appender.email.MaxEMails=10
log4j.appender.email.From=admin@mycompany.com
log4j.appender.email.To=me@mycompany.com
log4j.appender.email.Subject=My Module Error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

That’s it. Now you can get email alerts, without your mail box being blocked… ;-)