Re: large-scale log loading & parsing
Date: Sun, 16 Aug 2009 11:12:35 +0100
Message-ID: <6rudnZfT0MYLRhrXnZ2dnUVZ8hmdnZ2d_at_giganews.com>
"Docco" <adisapir_at_gmail.com> wrote in message
news:10caf2ce-9449-4d3c-b99e-87c37c10a0e6_at_t13g2000yqt.googlegroups.com...
> Hi,
> I'm looking for a solution for the following scenario:
> - System running on multiple servers (could be dozens)
> - Each server is running IIS and contains logs
>
> The solution is a way to easily provide analysis reports on a combined
> information from these logs. For example - "What's the IP Geo
> distribution between DateX and DateY"
>
> I am afraid that traditional methods (such as bulk-load everything
> into mssql) will not work because of the big load.
> I was also looking at solutions such as Hive (over Hadoop) but our
> environment is Win32 and I'm not sure it's the right path.
>
> So, I'm looking for ideas... Need to easily being able to load those
> logs and then easily analyze them
>
> Thanks!
>
> reply to adisapir [at] gmail dot com
Have you looked into any of the standard web analytics and clickstream analysis solutions? I suggest you consider buying one before you build it yourself.
-- David PortasReceived on Sun Aug 16 2009 - 12:12:35 CEST