Re: MYSQL Error 2013 load infile 15mln rec 6gb CSV

On 2014-10-24, Mladen Gogala <gogala.mladen_at_gmail.com> wrote:
> On Mon, 20 Oct 2014 23:11:45 +0100, Eric wrote:
>
>
>> idiot: someone who disagrees with you on a single point?
>
> Nope. Idiot is someone who can say that hints are not necessary because
> Postgres is not for profit.

I think what they said was that hints are unnecessary, and that they do not need to implement them in spite of that because Postgres is not for profit. If they are right about the first, the second is not idiotic, nor is it if they are convinced about the first even if it eventually turns out to be wrong. Because they are not for profit, they are not trying to compete with Oracle or anything else, and can look at things on a longer timescale.

Note: I have nothing to do with the Postgres team, I have never read a single line of the source code, I have a single instance of Postgres on my home server which I ignore as long as it runs because something else that I do "need" uses it. I'm just trying to start from a more open-minded position that you seem ever to have had, at least since you discovered there were no hints.

>> As for needing hints, how often, on any forum, is an OP told to step
>> above their actual question and state the goal behind it. Are you sure
>> that you are not just like them, assuming that the way to do something
>> in product X must exist in the same form in product Y.
>
> Nope. The hints are basically means to override the optimizer decision
> and require the execution plan to go differently than the optimizer has
> originally calculated. Insisting that optimizer decision should never be
> overridden means telling your users that you think that a computer
> program is smarter than them, when it comes to their data.

And what component of any other commercial software product do you need to override? How many of those give you a way to do it?

> State of
> affairs in technology is simply not there yet. Optimizer is right in the
> 99% of the cases, but for the remaining 1% percent, hints are needed,
> period. PostgreSQL is the only major database which doesn't have hints.
> And the idiot I quoted before is trying to make a technical inadequacy,
> something that PostgreSQL is missing, as opposed to all other major
> databases, into a phylosophical sticky point. The attitude is even more
> curious because there already are methods to change the execution plans,
> only the scope is session, not a single SQL. Unrelated to that, one of
> the main people designing Postgres is Bruce Momjian who works for the
> company which sells a commercial version of Postgres which supports
> hints. One of the replies I received on the mailing lists was "if you
> want hints, you will have to pay for them". Funny, isn't it?

So a commercial organisation made a commercial decision. Are you suggesting that the Postgres team is being used as a tool of that organisation? Then where is the project fork that would inevitably follow?

> Of course
> I will pay, only not to EnterpriseDB or other commercial versions of
> Postgres. If I have to pay for hints, it will be to the makers of Oracle,
> DB2 or MS SQL Server, who also sell databases which support hints. The
> makers of those databases do not have problems with hints.
> One of the points in the article by our former pastry baker is that
> "commercial databases have hints because DBAs want them". True, if you
> want me to use your software, you better make sure that it fulfills my
> needs. And that is why Postgres is remaining an obscure solution which I
> do not recommend to anyone, unless its developers change attitude and
> implement hints and some other things that users need.

>> OK, I use hints because they are the best way (or the only way) to
>> achieve a desired situation in Oracle, but I am quite prepared to
>> believe that they are not a universal answer.
>
> Your religion is not my problem.

Oh really!!! The verb "to believe" and its various derived words may be used frequently in connection with any or all religions, but they are not restricted to that usage (even "faith" is not restricted in this way), and for you to assume any such meaning on my part is actually rather insulting.

> There are people who are prepared to
> believe that they will go to heaven and be greeted by 69.98 virgins if
> they blow themselves up in a bus full of infidels. Dealing with the
> occasional optimizer hiccups is my problem. And I need hints in order to
> be able to do that. In other words, when I have SQL that needs to use
> specific index or access path, I must be able to tell that to the
> optimizer. If I cannot do that with the PostgreSQL, I will not use it,
> period.

> As for the "universal answer", it has already been found: "42". The only
> problem is what is the question. Hints are not the answer, hints are
> means of controlling the optimizer.

Universal answer to a particular and already mentioned question, thankyou. And of course there isn't one, but there may be a reasonable candidate other than the one you are so insistent on using.

> The question is whether users should
> have the ability to control the optimizer or should not have that
> ability. And the users themselves answered that question by
> overwhelmingly flocking to the databases which offer them such control,
> which is everything else except PostgreSQL.

See my question above about other software.
>
> I am just helping Postgres to remain obscure by constantly pointing out
> the attitude of developers who think that users do not need to have
> control of their optimizer. Such a condescending and rude attitude is
> hard to defend. It amounts to telling the users that they are too stupid
> to control the optimizer. I didn't accept that message too well

I don't see the message like that (obviously).

> and,
> judging by the size of Postgres conferences, neither have the vast
> majority of the other database users. I do believe in evolution. Ingres
> has lost to Oracle and Postgres is on its way to lose to Oracle too.

Evolution (in the biological sense) is not the march of progress, it is the march of increasing complexity, some examples of which survive for much longer than others due to various contingent circumstances. Often the survivors would have seemed unlikely when they first appeared. I suspect that this applies to software "evolution" as well.

Unfortunately people are more comfortable with the familiar, and are too inclined to see everything as "either/or", so the most popular "answer" is not necessarily the best.

Eric

-- 
ms fnd in a lbry