There are numerous Log Viewer / Log Analyzer tools in the market, which reflects the fact that standard log analysis functionality provided by most operating systems is insufficient to adequately address investigation of complicated software issues.
So, command-line oriented Linux/Unix OS provides powerful standard log analysis tools, such as grep and regular expressions. These are fairly sufficient for analysis of single Log Line data, e.g. calculating Transaction rate statistics. However, they fall short if we need e.g. to analyze correlation between log lines, i.e. recognition of logically linked lines. So, for the example above, if we would need to calculate statistics of system response time, we would need to first find the beginning and the end line for each transaction. Even in Linux, this task requires advanced scripting and eventually the task might prove too time-consuming to complete in reasonable time.
Windows uses essentially graphical user interface, but there is no standard graphical log analyzer, and standard text editors are intended for office use and only offer simple string search. Same applies to Mac OS, though (as it is based on Linux) it has an option of using a command-line interface as well. Linux environment can be simulated in Windows with the help of Cygwin software.
In our opinion, to select the Log Viewer best suitable for your tasks, you need to consider 5 main questions.
Question #1. What is the size of the log file you need to analyze. Even though in many cases software systems have some method of log rotation in place, at times Log Files can grow very big. Software professionals sometimes have to analyze files over 1 GB. Especially in Windows, handling of big files in not an easy task, so you should evaluate a tool’s performance for a log file of your size.
Question #2. Do you need to analyze Binary Files. Most tools only support logs in text format.
Question #3. Do you anticipate the need of merging multiple log files for analyzing on longer time intervals? E.g. for getting more reliable statistics. Not every tool supports this.
Question #4. Would you prefer a Graphical Analysis Tool. It is common knowledge that a picture can be worth a thousand words, but is it true in log analysis as well? The answer is yes. Each Log line is overloaded with information, and as we saw earlier, the number of lines in the log is usually too big to easily navigate between them visually – you can’t just keep that much information in your memory when it is scrolled on the screen. Graphics adds a new dimension to analysis, making it easier to see trends or just faster to find what you are looking for.
Adding graphical functionality to a Log Viewer is not an easy task. A a log file does not have a regular structure: usually log is the place where many independent processes write their messages, each might be using its own format. In comparison to such a well-structured file as e.g. spreadsheet, we can’t just plot one column against another.
One way to solve this problem is adding functionality of recognizing and analyzing of Log Patterns, which brings us to
Question #5. Would you need to search for Log Patterns and analyze them. Log pattern is a repeatable combination of keywords that may have different values associated with them.
Let us take an example. Any Web server log contains series of repeating HTTP GET requests and 200 OK (or other code) responses. Every request has an URI pointing to a certain page on the server. These 3 objects (GET, URI and 200 OK) form a natural pattern, and its occurrence as a function of time can be plotted, where the request URI and/or response code can be rendered as a caption for the graph point.
This example shows how powerful Log Analysis can become when log data processing using patterns is combined with graphical presentation.