Re: large-scale log loading & parsing

From: toby <toby_at_telegraphics.com.au>
Date: Mon, 17 Aug 2009 20:05:51 -0700 (PDT)
Message-ID: <4f3f329c-2cfb-4215-94a7-f04d273c39a1_at_c2g2000yqi.googlegroups.com>


On Aug 16, 6:12 am, "David Portas"
<REMOVE_BEFORE_REPLYING_dpor..._at_acm.org> wrote:
> "Docco" <adisa..._at_gmail.com> wrote in message
>
> news:10caf2ce-9449-4d3c-b99e-87c37c10a0e6_at_t13g2000yqt.googlegroups.com...
>
>
>
> > Hi,
> > I'm looking for a solution for the following scenario:
> > - System running on multiple servers (could be dozens)
> > - Each server is running IIS and contains logs
>
> > The solution is a way to easily provide analysis reports on a combined
> > information from these logs. For example - "What's the IP Geo
> > distribution between DateX and DateY"
>
> > I am afraid that traditional methods (such as bulk-load everything
> > into mssql) will not work because of the big load.
> > I was also looking at solutions such as Hive (over Hadoop) but our
> > environment is Win32 and I'm not sure it's the right path.
>
> > So, I'm looking for ideas... Need to easily being able to load those
> > logs and then easily analyze them
>
> > Thanks!
>
> > reply to adisapir [at] gmail dot com
>
> Have you looked into any of the standard web analytics and clickstream
> analysis solutions?

Like http://splunk.com just to name one at random.

> I suggest you consider buying one before you build it
> yourself.
>
> --
> David Portas
Received on Tue Aug 18 2009 - 05:05:51 CEST

Original text of this message