Logging

From logging primitives to drivers, to servers, to logs analysis platforms.

Think Drupal watchdog(), FIG PSR-3 standard, Monolog vs log4php, syslog vs gelf vs logstash, Graylog, fluentd, Flume, Rollbar, Loggly, and coopetitors.

How to add a server load graph to a Graylog dashboard

Submitted by Frederic Marand on

Graylog is, in the words of its creators, a tool to Store, search & analyze log data from any source, and it puts a lot of power in our hands to slice, dice, and generally combine, gather, and parse content from various sources, notably syslog and Gelf sources, as well as many file-type sources thanks to the Graylog Collector. Which means it makes it a snap to build event-oriented dashboards like the left part of this example, and even some event volume graphs like the topmost one on the right.

Graylog dashboard example on OSInet.fr

The problem: logging non-event information

This covers only logged events, which is fine overall, since Graylog is a log analysis platform, not a graph-oriented monitoring system like Munin / Cati / Ganglia et alii. However, in many cases, especially when building dashboards instead of performing some specific research, one may want to keep an eye on average system load, or other non-event information, if only to know when to switch one's attention to the monitoring system.

So how can one add system load information, which is not event-based, to a log dashboard like the three bottom-right graphs on the previous dashboard ?

Logging for MongoDB

Submitted by Frederic Marand on

One nice thing during Drupal 7/8 development is the ability, thanks to the devel module, to get a list of all SQL queries ran on a page. As I've been working quite a bit on MongoDB in PHP recently, I wondered how to obtain comparable results when using MongoDB in PHP projects. Looking at the D7 implementation, the magic happens in the Database class:

<?php
// Start logging on the default database.
define(DB_CHANNEL, 'my_logging_channel');
\Database::startLog(DB_CHANNEL);

// Get the log contents, typically in a shutdown handler.
$log = \Database::getLog(DB_CHANNEL);
?>

With DBTNG, that's all it takes, and devel puts it to good use UI-wise. So is there be an equivalent mechanism in MongoDB ? Of course there is !

Rethinking watchdog(): logging in Kohana 3

Submitted by Frederic Marand on

Continuing this exploration of logging solutions used in various projects, let's look at logging in Kohana 3.

Kohana 3.3 Logging - bundled classes While Monolog and log4php share a mostly common logging model of a frontal Logger object instantiated as many times as needed to supply different logging channels, in which log events are Processed/Filtered then written out by Handlers/Writers, Kohana builds upon a simpler model, which can be summarized by three patterns:

  • Singleton: there is only one instance of the Kohana Log
  • Observer: Log_Writer instances are attached (and detached) to(/from) the logger instance and handle events they are interested in based on their own configuration. Much like a Drupal hook, all writer instances receive each Log event
  • Delegation: the Log exposes a write() to trigger the buffered writing, but does not implement it itself, but delegates to the Log_Writer objects to perform it. Buffered logging control is a Log property, not a Log_Writer property.

Rethinking watchdog(): Monolog vs log4php

Submitted by Frederic Marand on

Beyond Monolog, other packages provide advanced logging services. Apache log4php is another well-known logging solution, used (among others) by CMS Made Simple, SugarCRM, and vTiger CRM.

It is based on the famous log4j package from the Java world, and from uses of this package I have seen on customer sites, I feel that it carries a lot of useless baggage, and is - in my opinion - significantly less of a good match than Monolog for Drupal 8.

Monolog vs log4php : equivalences

There is some degree of equivalence between the Monolog and log4php components:

Purpose Monolog log4php Notes
Log an event Logger Logger Very similar
Store an event Handler Appender both can be chained, group, control bubbling (Monolog) / filtering (log4php)
Format an event representation Formatter Layout log4php layouts can format a group of events, Monolog formatters format an individual event
Massage event data Processor Renderer Not so similar. Monolog processors will often add extra data, while log4php Renderers are typically used to format non-string events as strings.

Rethinking watchdog(): Monolog architecture

Submitted by Frederic Marand on

I've been discussing Monolog in Drupal events (DrupalCamp Lyon, DevDays Barcelona) as a possible alternative to the legacy Drupal watchdog() service for quite some time, but never took the time to explain it in writing, and the feature freeze date is looming ahead, so since I'm taking part in th Gent code sprint, and code has been starting to take shape here, here, and there, here is an overview of the Monolog classes.

The diagram below is a simplified version of the Monolog architecture. It includes all classes and interfaces, but only the most significant methods, no constants, and none of the non-bundled classes and interfaces upon which some of the builtins depend.

simplified class diagram for Monolog 1.1 (SVG rendering)