Re: Slightly OT: ETL Tools
Date: Mon, 9 Jan 2012 11:34:10 -0800 (PST)
Message-ID: <1326137650.29339.YahooMailNeo_at_web121004.mail.ne1.yahoo.com>
Hi Greg,
No, I think you hit on the points that seem to be very important to the DBA's and Developers here at my company. IF something does go wrong or DOES require change, currently, it's comfortably "in SQL we trust". The concerns of introducing an ETL tool is a common sore point to any DBA/Developer who has managed an environment with an ETL tool where you have sent the SQL to the ETL folks, they have shrugged their shoulders and said, "I have no idea where that is coming from, can't be us!" This is commonly after it has been clearly traced back to the ETL tool and it takes them quite some time before they've had to come back and say, "Oops.. found it." For me, this has nothing to do with pointing fingers, but with the ability to quickly find and correct issues. I know I'm one to sometimes complain about the complexity of my own environment without an ETL tool, but even now, the areas I can't fix when people say, "I want it fixed now, can't you DBA's do anything?!" is from an interface that performs tasks outside the database! The DBA's and developers have a lot more control over the moving parts right now- can't often say that with an ETL tool.
I know some ETL tools do that better than others, so if anyone can offer up some names to add to Greg's valuable info, I'll take them! :)
Kellyn Pot'Vin
Sr. Database Administrator and Developer
DBAKevlar.com
From: Greg Rahn <greg_at_structureddata.org> To: kellyn.potvin_at_ymail.com
Cc: "oracle-l_at_freelists.org" <oracle-l_at_freelists.org> Sent: Monday, January 9, 2012 12:23 PM
Subject: Re: Slightly OT: ETL Tools
For me, I see just two requirements of any ETL product, purchased or engineered, in today's world that interacts with data in an RDBMS: 1) The output must be SQL. This means that the ETL product does code generation only and the database does the actual work. This way your ETL scales as your database scales. 2) The product must be metadata driven. Any ETL code that is not metadata driven is costly to maintain and becomes brittle and problematic when changes are required (and their will be changes).
I realize that this does not specifically address any of the points you have asked, but I think they are the most important technical points out there when dealing with large systems or systems that can grow up to be large.
On Mon, Jan 9, 2012 at 10:52 AM, Kellyn Pot'vin <kellyn.potvin_at_ymail.com> wrote:
Hey Listers!
>I've been tasked with a new data warehouse project and one of the steps is to decide if we wish to continue all our transformations and build steps through SQL or if we would be better off utilizing an ETL tool. I have documented, as well as my manager, the pros and cons of going with an ETL, but I'm hitting up a number of resources on what ETL tools are preferred by different groups.
>Developers and DBA's, from your POV, what ETL tools do you think are the best?
>I'm looking for the following:
>1. What was the product and how long did you work with it?
>2. In what capacity you supported the product, (i.e. as a DBA or developer) and/or did you perform the tasks as an ETL administrator or supported the technical peer who did perform these functions for the business?
>3. What were the best features/ limiting drawbacks of the product?
>
>Questions may seem involved, but feel free to just slap a few lines of info together and I would really appreciate any feedback you can offer!
>Thanks!
--
Regards,Greg Rahn | blog | twitter | linkedin
-- http://www.freelists.org/webpage/oracle-lReceived on Mon Jan 09 2012 - 13:34:10 CST