Event log analysis for the big leagues
Wow: Christina has joined Addamark, makers of the Log Management System (LMS) used by Agilent, Walmart.com, and Terra Lycos to store, analyze, and manage jillions of lines of event log data. My lovely wife is now Addamark’s director of systems consulting, which merges her executive management experience with her ability to repartition disks in the middle of a demo — without having done it before.
To a blogger like me, the appeal of LMS is obvious: It lets you obsess over the story told not on your site, but in your logs: Who’s visiting, and from where? What are they doing, and why, and how has that changed over time? Are we doing something wrong? But instead of a few thousand events a day in their logs, these companies have millions, and are headed for billions.
LMS is a SQL database engine extensible through embedded Perl subroutines, but one specialized and optimized for read-only log data as opposed to, say, financial transaction records. And it’s designed to be run by a Unix admin, rather than requiring a DBA specialist. Having tried to hire a DBA not too long ago, I can appreciate that feature.
Addamark Log Management System
|
“Ultra-compressed” means Addamark packs data many times denser in LMS than in a typical warehouse built atop Oracle or gzip, which are more general-purpose solutions.
All of this is easy to demand, hard to deliver. I’m really glad to see people like Christina and Adam Sah - one of the leading engineers behind HotBot’s launch in 1996 - working together on the solution.
Pop quiz: How much disk space would LMS need to handle all Radio logs through 2006? I don’t know the answer, I just want to see how they would figure it out. 