large-scale log loading & parsing
From: Docco <adisapir_at_gmail.com>
Date: Sun, 16 Aug 2009 02:02:14 -0700 (PDT)
Message-ID: <10caf2ce-9449-4d3c-b99e-87c37c10a0e6_at_t13g2000yqt.googlegroups.com>
Hi,
I'm looking for a solution for the following scenario:
- System running on multiple servers (could be dozens)
Date: Sun, 16 Aug 2009 02:02:14 -0700 (PDT)
Message-ID: <10caf2ce-9449-4d3c-b99e-87c37c10a0e6_at_t13g2000yqt.googlegroups.com>
Hi,
I'm looking for a solution for the following scenario:
- System running on multiple servers (could be dozens)
- Each server is running IIS and contains logs
The solution is a way to easily provide analysis reports on a combined information from these logs. For example - "What's the IP Geo distribution between DateX and DateY"
I am afraid that traditional methods (such as bulk-load everything into mssql) will not work because of the big load. I was also looking at solutions such as Hive (over Hadoop) but our environment is Win32 and I'm not sure it's the right path.
So, I'm looking for ideas... Need to easily being able to load those logs and then easily analyze them
Thanks!
reply to adisapir [at] gmail dot com Received on Sun Aug 16 2009 - 11:02:14 CEST