Greg Pavlik

Subscribe to Greg Pavlik feed
Welcome to the blog of Greg Pavlik, software technologist and frustrated adventurer. Currently, I am working on technologies related to processing very large data sets. This blog will contain a mix of commentary on technology, our industry and unrelated topics that interest me.Greg Pavlikhttp://www.blogger.com/profile/02076590604248408230noreply@blogger.comBlogger247125
Updated: 16 hours 12 min ago

The river floes break in spring...

Wed, 2016-05-25 19:37

Alexander Blok
 The river floes break in spring...
March 1902
translation by Greg Pavlik 


The river floes break in spring,
And for the dead I feel no sorrow -
Toward new summits I am rising,
Forgetting crevasses of past striving,
I see the blue horizon of tomorrow.

What regret, in fire and smoke,
What agony of Aaron’s rod,
With each hour, with each stroke -
Or instead - the heavens’ gift stoked,
From the Bush of Moses, the Mother of God!

Original:

Весна в реке ломает льдины,
И милых мертвых мне не жаль:
Преодолев мои вершины,
Забыл я зимние теснины
И вижу голубую даль.

Что сожалеть в дыму пожара,
Что сокрушаться у креста,
Когда всечасно жду удара
Или божественного дара
Из Моисеева куста!
 
 Март 1902

Why I am a Dostoevskyan Humanist

Sun, 2016-05-01 17:11
An explanation in 5 parts, by reference to the works of those who were not.*

'Lo! I show you the Last Man.

"What is love? What is creation? What is longing? What is a star?" -- so asks the Last Man, and blinks.

The earth has become small, and on it hops the Last Man, who makes everything small. His species is ineradicable as the flea; the Last Man lives longest.

"We have discovered happiness" -- say the Last Men, and they blink.

They have left the regions where it is hard to live; for they need warmth. One still loves one's neighbor and rubs against him; for one needs warmth.

Turning ill and being distrustful, they consider sinful: they walk warily. He is a fool who still stumbles over stones or men!

A little poison now and then: that makes for pleasant dreams. And much poison at the end for a pleasant death.

One still works, for work is a pastime. But one is careful lest the pastime should hurt one.

One no longer becomes poor or rich; both are too burdensome. Who still wants to rule? Who still wants to obey? Both are too burdensome.

No shepherd, and one herd! Everyone wants the same; everyone is the same: he who feels differently goes voluntarily into the madhouse.

"Formerly all the world was insane," -- say the subtlest of them, and they blink.

They are clever and know all that has happened: so there is no end to their derision. People still quarrel, but are soon reconciled -- otherwise it upsets their stomachs.

They have their little pleasures for the day, and their little pleasures for the night, but they have a regard for health.

"We have discovered happiness," -- say the Last Men, and they blink.'
Friedrich Nietzsche: Thus Spoke Zarathustra



The Body of the Dead Christ in the Tomb, Hans Holbein

Now, did He really break the seal
And rise again?
We dare not say….
Meanwhile, a silence on the cross
As dead as we shall ever be,
Speaks of some total gain or loss,
And you and I are free
Auden, Friday’s Child

“Wherever an altar is found, there is civilization."
Joseph de Maistre

“All actual life is encounter.”
Martin Buber, I and Thou

* model for composition stolen gratuitously from an online challenge.

Something Amiss

Thu, 2016-03-17 00:02
Looks like this curious non-review of the novel Laurus seems to have been referring to "Brahmins" as "Brahman" - I suppose republished to correct the mistake:
​Russian Brahman by Alan Jacobs | Articles | First Things
www.firstthings.com/article/2016/04/russian-brahman
First Things
Russian Brahman. by Alan Jacobs April 2016. Laurus by eugene vodolazkin translated by lisa hayden oneworld, 384 pages, $24.99. Eugene Vodolazkin's ...
​Russian Brahmin by Alan Jacobs | Articles | First Things
www.firstthings.com/article/2016/04/russian-brahmin
First Things
Russian Brahmin. by Alan Jacobs April 2016. Laurus by eugene vodolazkin translated by lisa hayden oneworld, 384 pages, $24.99. Eugene Vodolazkin's ...

Whatever his grasp of Hindu concepts, it's obvious Jacobs knows little to nothing about the tradition of Russian yurodivy, which makes this review overall kind of silly at best. Interested readers can refer to the hagiographies of Xenia of Petersburg or Feofil of the Kiev Caves Lavra to become acquainted with some of the conceptual background to the novel, both published by the monastery press in Jordanville, NY in English. As a complement the Pavel Lungin movie Ostrov is worth watching carefully - the film is based partly on Feofil, though like the life of St Xenia, it explores the theme of vicarious repentance. (It was not until the third time I saw the film that I fully grasped it - the visuals are stunning and in many respects a distraction.)

All of that aside, what continues to trouble me in general is the fact that most of the reviews of Laurus that I've seen have been oriented toward theological critiques - endorsements or arguments revolving around the reviewer's reading of what the author might want us to think about religion. And yet it is obvious that Vodolazkin did not write a religious apologetic (Jacobs invokes Karamazov, which is simultaneously a religious argument and a humanistic work - but Laurus is anything but the former). Laurus deserves a review as a work of notable - even great - world literature: which is to say first and foremost an exploration of what Vodolazkin is attempting to accomplish as a writer and what that has produced as a work of literature. The lack of serious analysis is particularly puzzling given the devices Vodolazkin uses to deal with language, identity, personality, relationship, and - yes - time. We could do with a few less sermons and a bit more thought.

Nativity

Thu, 2015-12-24 09:24
Shadows flicker against the wall
within the cave it is perpetually night
(I find my vision gets dimmer with age
- when we are alleged to see more sharply -
in the low light of a single candle flame
it is getting much harder to read
year by year)
there is a form I barely am able to perceive.
I wonder if it is better here than the open air
where my eyes would surely be closed against the sun
where all forms find their origin in the one.

2015

Brazil

Sun, 2015-12-20 23:11
Blown away to get my purple belt in Brazilian Jiu Jitsu from 10th Planet black belt Alex Canders.


Like a white stone in the depths of a well...

Tue, 2015-10-13 13:26
Like a white stone in the depths of a well...
Anna Akhmatova
working translation by Greg Pavlik

Like a white stone in the depths of a well
within me there lies one memory.
I can not - and do not - want to expel
this, my greatest joy and my agony.

I think that anyone who closely looks
can see this recollection reads
as harrowing sadness in a tragic book -
a warning, and a sign of need.

I know the gods forever strive
to wreck the body but cannot touch the mind;
assurance that you will forever live
as a memory I can’t leave behind.

Original:
Как белый камень в глубине колодца,
Лежит во мне одно воспоминанье.
Я не могу и не хочу бороться:
Оно - веселье и оно - страданье.

Мне кажется, что тот, кто близко взглянет
В мои глаза, его увидит сразу.
Печальней и задумчивее станет
Внимающего скорбному рассказу.

Я ведаю, что боги превращали
Людей в предметы, не убив сознанья,
Чтоб вечно жили дивные печали.
Ты превращен в мое воспоминанье.

Requiem Fragment

Sun, 2015-10-11 16:59
Requiem Fragment
Anna Akhmatova
translation by Greg Pavlik

10 Crucifixion

      Do not lament me, O Mother,
      seeing me in the tomb.


The angelic chorus glorified the hour of eternity,
when the heavens convulsed in a river of fire:
He cried to His Father “Eli, Eli, lama sabachthani?”
And to His Mother, he spoke: “Do not lament me”…

1938

Mary Magdalena writhed, and wept
as the beloved John froze like stone (or salt).
Where the Mother stood in silence -
no one dared to look.

1940

Translator notes. I chose to translate this poem much less loosely than my interpretative translation of Akhmatova’s Lot’s Wife – there Russian readers will recognize that the final stanza is essentially my own poem (with an explicit reference to the unrelated work of Scott Cairns, in fact).

Instead, this section of Requiem is much closer to the original (1), with only minimal augmentation. Nonetheless, it is also very different from Akhmatova’s in certain critical respects. Requiem itself is a difficult and evocative work – its melancholy is inseparable from the suffering of both Akhmatova herself and the Soviet people under Stalin. In some sense, I have abandoned this context in my translation. Educated Russian readers would have recognized Akhmatova’s work as using the imagery of the hymnography of the Paschal Nocturnes, the final liturgical setting of Great and Holy Saturday chanted before the entombed body of the dead Christ, which includes a deeply moving dialogue with his Mother. (2)

I have chosen to deepen the liturgical elements of the poem and play off themes that recur in and around the Lenten Triodion. The text of the Slavonic service is rendered here in English as it is commonly used in American parishes within the Russian Orthodox tradition. Second, I use the Hebrew directly in quotation from the Psalter, emphasizing its position as a liturgical prayer. While the dialogic element from the Nocturnes service is repeated, maintaining the liturgical connection, here the address to the Mother is clearly one of human filial affection.

The river of fire is evocative of the image God as “consuming fire”, which, St Isaac says is experienced as bliss by the pure in heart. The second stanza – and I do not believe there is any intention at all in the original to do this – also points back to the story of Lot and Sodom and implicitly re-invokes the image of fire. Akhmatova makes no association with salt. I have tried intentionally not to recall the poetics of Stabat Mater in the final lines.

(1) Original Russian:

10
РАСПЯТИЕ

   Не рыдай Мене, Мати,
   во гробе зрящия.

Хор ангелов великий час восславил,
И небеса расплавились в огне.
Отцу сказал: "Почто Меня оставил!"
А матери: "О, не рыдай Мене..."

1938

Магдалина билась и рыдала,
Ученик любимый каменел,
А туда, где молча Мать стояла,
Так никто взглянуть и не посмел.

1940, Фонтанный Дом

(2) From the Eastern Orthodox Liturgical Service of Paschal Nocturnes:

Do not lament me, O Mother, seeing me in the tomb, the Son conceived in the womb without seed, for I shall arise and be glorified with eternal glory as God. I shall exalt all who magnify thee in faith and in love.

Glory to Thee, our God, glory to Thee!

I escaped sufferings and was blessed beyond nature at Thy strange birth, O Son, who art without beginning. But now, beholding Thee, my God, dead and without breath, I am sorely pierced by the sword of sorrow. But arise, that I may be magnified.

Glory to Thee, our God, glory to Thee!

By my own will, the earth covers me, O Mother, but the gatekeepers of hell tremble at seeing me clothed in the blood-stained garments of vengeance; for when I have vanquished my enemies on the cross, I shall arise as God and magnify thee.

Glory to the Father and to the Son and to the Holy Spirit, now and ever and unto ages of ages. Amen.

Let creation rejoice, let all born on earth be glad, for hateful hell has been despoiled, let the women with myrrh come to meet me, for I am redeeming Adam and Eve and all their descendants, and on the third day shall I arise.

Lot's Wife

Mon, 2015-09-28 11:57
Lot's Wife
by Anna Akhmatova
an interpretive translation by Greg Pavlik
The righteous Lot heard the voice of God
As if coming from the distant and black mountains.
But his wife,
She saw what was until yesterday her blessing.
Under the beautiful spires of Sodom,
Where she sang spinning cloth -
The empty window of the room,
In which her children were born.
She looked – and her pain died with her,
For she could look no more:
Her body translucent salt,
Her feet joined the earth.
Who will mourn for Marah?
An insignificant role in a grand saga -
Yet my conscience cannot forget
The one who gave her life for a fleeting glance.
-----
Original poem
И праведник шел за посланником Бога,
Огромный и светлый, по черной горе.
Но громко жене говорила тревога:
Не поздно, ты можешь еще посмотреть
На красные башни родного Содома,
На площадь, где пела, на двор, где пряла,
На окна пустые высокого дома,
Где милому мужу детей родила.
Взглянула – и, скованы смертною болью,
Глаза ее больше смотреть не могли;
И сделалось тело прозрачною солью,
И быстрые ноги к земле приросли.
Кто женщину эту оплакивать будет?
Не меньшей ли мнится она из утрат?
Лишь сердце мое никогда не забудет
Отдавшую жизнь за единственный взгляд.

Ancestor Worship

Sat, 2015-08-08 08:25
Some profound lessons in how to be human that we can learn from our Confucian friends


Fascinating Lives

Fri, 2015-08-07 18:38
There is something, I think, admirable in a quiet life: care for family, constructive participation in community, hard work. But there are times and places (perhaps all times, but not all places?) where simply attending to the simple things of life becomes a kind of impossibility: whether for psychological or moral reasons. I was reflecting on two persons recently who have struck me by not only their intellectual genius but also by the sheer force by which they pushed against the norm, one for reasons of psychology and one for reasons of morality.

Yukio Mishima: narcissist, political fanatic, suicide. And one of Japan's greatest novelists. I recently completed the Sea of Fertility tetralogy, which traces the life of Shigekuni Honda from youth to retirement as a wealthy attorney, centered around what Honda believes are the successive reincarnations of his friend Kiyoaki Matsugae: as a young rightist, a Thai princess and an orphan. The most powerful of the four novels, in my opinion is the second: Runaway Horses. The book seems to rebuke the militant nationalism of Japanese reactionaries, though ironically enough Mishima himself ends his own life under the banner of a similar ideology. Mishima's fascinating portrait of an inherent dark side of youth - a taming of a deep inhumanism - so to speak, comes through almost all the novels, but most strongly in the last. This echoes a theme he developed in The Sailor Who Fell From Grace with the Sea, though I can think of few works that more strongly explore this theme than the Lord of the Flies. In any case, Mishima is masterful in exploring aberrant developmental psychology - even as he, himself, seems to have been stricken with his own disordered personality.

Maria Skobtsova: atheist, symbolist poet, Bolshevik revolutionary - and a renegade nun arrested for helping Jews in Paris by the Gestapo, she allegedly died by taking the place of a Jewish woman being sent to death. Jim Forrest provides a useful overview of her life - unlikely most lives of a Christian saints, this is no hagiography: it is a straightforward story of life. At the same time, we see a life transformed by a dawning realization that self-denial is a path to transformation -

"The way to God lies through love of people. At the Last Judgment I shall not be asked whether I was successful in my ascetic exercises, nor how many bows and prostrations I made. Instead I shall be asked did I feed the hungry, clothe the naked, visit the sick and the prisoners. That is all I shall be asked. About every poor, hungry and imprisoned person the Savior says ‘I': ‘I was hungry and thirsty, I was sick and in prison.’ To think that he puts an equal sign between himself and anyone in need. . . . I always knew it, but now it has somehow penetrated to my sinews. It fills me with awe."

And despite a life dedicated to service, she remained an acute intellectual, a characteristic of so many Russian emigres in Paris. This too reflected her view that redemption and suffering where intertwined - my favorite piece On the Imitation of the Mother of God
- draws this out beautifully.


Aporias

Sat, 2015-06-20 11:03
1.
Actually
He likes word games
   Diffident
In the way they circle about
   Starlings in flight
   Or Seraphim.

2.
With sweeping gesture
   Left to right
Hands hang with head
Under the pressing of the sun:
   Weight of doubt
   Or will.

3.
Against open air, tumultuous sea
   Turtle green
The division is nowhere more evident
Where sand meets froth
   Stark, blinding glare
Wind balmed
   Until night.

4.
Cleanliness,
   Next to godliness
The echo of countless schoolmarms
Chiding, chilling - without regret
   Yeah, rather,
   Motherhood.

What The World Needs More Of

Fri, 2015-06-19 15:07
The interview with these two kids - Chris and Camryn Singleton - is available on BBC, but I wanted to pull out this remarkable commentary in a related article:

"People are hurting in Charleston. But for the hundreds who packed into the gymnasium at the Goose Creek High School, it was also a reminder of the importance of love.

Sharonda Singleton coached the girls' athletics team here. As her photo rested on an easel on the polished floors in the vast sports hall, her friends and family paid tribute. Speaking for the first time since the deadly attack on the AME church where she worshipped, Sharonda's two children, Chris and Camryn, told me they forgive the man who killed her.

 "We already forgive him and there's nothing but love from our side of the family," Chris told me.

Many will find this incomprehensible. Charleston is often called the Holy City for the number of churches it is home to, and the role religion plays here. For some, like Chris and Camryn, unwavering faith is the only way to turn such a devastating loss into something positive."

This immediately brought to mind the sayings of Fr Zosima in Dostoevsky's Brothers Karamazov, which not to many years ago were the source of a kind of epiphany for me that in a sense reoriented by own thinking:

"Strive to love your neighbor actively and indefatigably. In as far as you advance in love you will grow surer of the reality of God and of the immort
ality of your soul. If you attain to perfect self-forgetfulness in the love of your neighbor, then you will believe without doubt, and no doubt can possibly enter your soul. This has been tried. This is certain.

Above all, avoid falsehood, every kind of falsehood, especially falseness to yourself. Watch over your own deceitfulness and look into it every hour, every minute. Avoid being scornful, both to others and to yourself. What seems to you bad within you will grow purer from the very fact of your observing it in yourself. Avoid fear, too, though fear is only the consequence of every sort of falsehood. Never be frightened at your own faint-heartedness in attaining love. Don't be frightened overmuch even at your evil actions. I am sorry I can say nothing more consoling to you, for love in action is a harsh and dreadful thing compared with love in dreams. Love in dreams is greedy for immediate action, rapidly performed and in the sight of all. Men will even give their lives if only the ordeal does not last long but is soon over, with all looking on and applauding as though on the stage. But active love is labor and fortitude, and for some people too, perhaps, a complete science. But I predict that just when you see with horror that in spite of all your efforts you are getting farther from your goal instead of nearer to it—at that very moment I predict that you will reach it and behold clearly the miraculous power of the Lord who has been all the time loving and mysteriously guiding you."

....


"At some thoughts one stands perplexed, especially at the sight of men's sin, and wonders whether one should use force or humble love. Always decide to use humble love. If you resolve on that once for all, you may subdue the whole world. Loving humility is marvelously strong, the strongest of all things, and there is nothing else like it." 

....

"“Remember particularly that you cannot be a judge of anyone. For no one can judge a criminal until he recognizes that he is just such a criminal as the man standing before him, and that he perhaps is more than all men to blame for that crime. When he understands that, he will be able to be a judge. Though that sounds absurd, it is true. If I had been righteous myself, perhaps there would have been no criminal standing before me. If you can take upon yourself the crime of the criminal your heart is judging, take it at once, suffer for him yourself, and let him go without reproach. And even if the law itself makes you his judge, act in the same spirit so far as possible, for he will go away and condemn himself more bitterly than you have done. If, after your kiss, he goes away untouched, mocking at you, do not let that be a stumbling-block to you. It shows his time has not yet come, but it will come in due course. And if it come not, no matter; if not he, then another in his place will understand and suffer, and judge and condemn himself, and the truth will be fulfilled. Believe that, believe it without doubt; for in that lies all the hope and faith of the saints.”


This time, Chris and Camryn have moved me beyond words by living this reality.

Addendum/edit: more of this humbling love on display

3 film non-meme

Sun, 2014-09-21 16:09
Riffing off previous post - was discussing with my wife last evening what we thought the three best "recent" films we had seen were. Here's my list:

1) Jia Zhangke's A Touch of Sin.

Reason: this is a powerful, powerful film that explores the effects of radical individualism, and economic inequality and of the overturning of normal, local, rooted communities. Banned by the Chinese government, it is as much a critique of the values of neoliberalism globally as it is of the current Chinese economic experiment.

2) Alejandro González Iñárritu's Biutiful.

Reason: a moving exploration of responsibility and ethics in the face of poverty, hopelessness and impending death. What do we make of the human spirit and our obligations to each other - and our obligations in the face of The Other?  Javier Bardem was birthed for this role - fantastic acting.

3) Pavel Lungin's The Island.

Reason: who is guilty before whom and for what? Take a director of Jewish background, give him a story that is loosely inspired by a hagiography of the fool-for-Christ Feofil of the Kieven Caves, and cast a retired-rock-star-current-recluse (Pyotr Mamonov) as a Orthodox monastic in the far north of Russia, and I would have quite low expectations for the outcome. What Lungin produced is instead not only his best film but I think one of the best films of the last 20 years.

Top 10 Book Meme

Tue, 2014-09-16 11:48
What books have most impacted me? I picked books I have returned to over and over. Yes, I know this is solipsistic to publish, but its a fascinating thing to think through. I'm sure the list will not look right in a few months anyway. But here I go...

1 The Brothers Karamazov, Fyodor Dostoyevsky
Reason: the best book ever written about the human condition.

2 Iob, LXX
Reason: bad things happen to good people, quite often.

3 I Am a Cat, Natsume Soseki
Reason: comedy is good for the soul. This is the funniest book I've ever read. Cheating here, but I'd probably add Gogol as a next choice.

4 The Symposium, Plato
Reason: love. And I'm an only partially reconstructed platonist.

5 Demons, Fyodor Dostoyevsky
Reason: explains a big part of the 20th century. Makes 1984 look like crude propaganda.

6 Also Spracht Zarathustra, Friedrich Nietzsche
Reason: Nietszche saw the enormity of the modern project clearly.

7 Life of Moses, Gregory of Nyssa
Reason: before structuralism, post structuralism, semiotics, and deconstruction, there was Gregory of Nyssa. And apokatastasis.

8 For the Time Being, WH Auden
Reason: aside from the fact that Auden is the best English language poet, this is a deeply moving meditation on Christmas in the anglophone experience. Read it several times each winter.

9 The Divine Comedy, Dante Alighieri
Reason: great art meets allegory meets beauty in the search for meaning. Honestly, stuck with only one book this might be it.

10 Faust, Johann Wolfgang von Goethe
Reason: we are all compromised to one degree or another.

Silence

Sat, 2014-07-26 11:26
Silence. Sometimes sought after, but in reality almost certainly feared - the absence of not just sound but voice. Silence is often associated with divine encounter - the neptic tradition of the Philokalia comes to mind - but also and perhaps more accurately with abandonment, divine or otherwise. I recently read Shusaku Endo's Silence, a remarkable work, dwelling on the theme of abandonment in the context of the extirpation of Kakure Kirishitan communities in Tokagawa Japan. Many resilient families survived and eventually came out of hiding in the liberalization in the mid-19th century, but the persecutions were terrible. Their story is deeply moving (sufficiently so that over time I find myself drawn to devotion to the image of Maria-Kannon). Endo's novel was not without controversy but remains one of the great literary accomplishments of the 20th century.

In fact, the reason for this post is a kind of double entendre on silence: the relative silence in literate western circles with respect to Japanese literature of the past century. Over the last month, I realized that virtually no one I had spoken with had read a single Japanese novel. Yet, like Russia of the 19th century, Japan produced a concentration of great writers and great novelists in the last 20th century that is set apart: the forces of of profound national changes (and defeat) created the crucible of great art. That art carries the distinctive aesthetic sense of Japan - a kind of openness of form, but is necessarily the carrier of universal, humanistic themes.

Endo is a writer in the post war period - the so-called third generation, and in my view the last of the wave of great Japanese literature. Read him. But don't stop - perhaps don't start - there. The early 20th century work of Natsume Soseki are a product of the Meiji period. In my view, Soseki is not only a father of Japenese literature but one of the greatest figures of world literature taken as a whole - I am a Cat remains one of my very favorite novels. Two troubling post-war novels by Yukio Mishima merit attention - Confessions of a Mask and the Sailor Who Fell From Grace with the Sea, both I would characterize broadly as existential masterpieces. The topic of identity in the face of westernization is also a moving theme in Osamu Dazai's No Longer Human. I hardly mean this as a complete survey - something in any case I am not qualified to provide -just a pointer toward something broader and important.

My encounter with contemporary Japanese literature - albeit limited - has been less impactful (I want to like Haruki Murakami in the same way I want to like Victor Pelevin, but both make me think of the distorted echo of something far better). And again like Russia, it is difficult to know what to make of Japan today - where its future will lead, whether it will see a cultural resurgence or decline. It is certain that its roots are deep and I hope she finds a way to draw on them and to flourish.


Spark: A Discussion

Wed, 2014-07-23 09:36
A great presentation, worth watching in its entirety.

With apologies to my Hadoop friends but this is good for you too.

Exactly Wrong

Mon, 2014-07-21 09:58
I normally avoid anything that smacks of a competitive discussion on what I consider to be a space for personal reflection. So while I want to disclose the fact that I am not disinterested in the points I am making from a professional standpoint, my main interest is to frame some architecture points that I think are extremely important for the maturation and success of the Hadoop ecosystem.

A few weeks back, Mike Olson of Cloudera spoke at Spark Summit on how Spark relates to the future of Hadoop. The presentation can be found here:

http://youtu.be/8kcdwnbHnJo

In particular I want to draw attention to the statement made at 1:45 in the presentation that describes Spark as the "natural successor to MapReduce" - it becomes clear very quickly that what Olson is talking about is batch processing. This is fascinating as everyone I've talked to immediately points out one obvious thing: Spark isn't a general purpose batch processing framework - that is not its design center. The whole point of Spark is to enable fast data access and interactivity.
 
The guys that clearly "get" Spark - unsurprisingly - are DataBricks. In talking with Ion and company, it's clear they understand the use cases where Spark shines - data scientist driven data exploration and algorithmic development, machine learning, etc. - things that take advantage of the memory mapping capabilities and speed of the framework. And they have offered an online service that allows users to rapidly extract value from cloud friendly datasets, which is smart.

Cloudera's idea of pushing SQL, Pig and other frameworks on to Spark is actually a step backwards - it is a proposal to recreate all the problems of MapReduce 1: it fails to understand the power of refactoring resource management away from the compute model. Spark would have to reinvent and mature models for multi-tenancy, resource managemnet, scheduling, security, scaleout, etc that are frankly already there today for Hadoop 2 with YARN.

The announcement of an intent to lead an implementation of Hive on Spark got some attention. This was something that I looked at carefully with my colleagues almost 2 years ago, so I'd like to make a few observations on why we didn't take this path then.

The first was maturity, in terms of the Spark implementation, of Hive itself, and Shark. Candidly, we knew Hive itself worked at scale but needed significant enhancement and refactoring for both new features on the SQL front and to work at interactive speeds. And we wanted to do all this in a way that did not compromise Hive's ability to work at scale - for real big data problems. So we focused on the mainstream of Hive and the development of a Dryad like runtime for optimal execution of operators in physical plans for SQL in a way that meshed deeply with YARN. That model took the learnings of the database community and scale out big data solutions and built on them "from the inside out", so to speak.

Anyone who has been tracking Hadoop for, oh, the last 2-3 years will understand intuitively the right architectural approach needs to be based on YARN. What I mean is that the query execution must - at the query task level - be composed of tasks that are administered directly by YARN. This is absolutely critical for multi-workload systems (this is one reason why a bolt on MPP solution is a mistake for Hadoop - it is at best a tactical model while the system evolves).  This is why we are working with the community on Tez, a low level framework for enabling YARN native domain specific execution engines. For Hive-on-Tez, Hive is the engine and Tez provides the YARN level integration for resource negotiation and coorindation for DAG execution: a DAG of native operators analogous the the execution model found in the MPP world (when people compare Tez and Spark, they are fundamentally confused - Spark could be run on Tez for example for a much deeper integration with Hadoop 2 for example). This model allows the full range of use cases from interactive to massive batch to be administered in a deeply integrated, YARN native way.

Spark will undoubtedly mature into a great tool for what it is designed for: in memory, interactive scenarios - generally script driven - and likely grow to subsume new use cases we aren't anticipating today. It is, however, exactly the wrong choice for scale out big data batch processing in anything like the near term; worse still, returning to a monolithic general purpose compute framework for all Hadoop models would be a huge regression and is a disastrously bad idea.

Dependent Rational Animals

Sun, 2014-07-20 17:32
I wanted to briefly comment on Alisdair MacIntyre's lectures collected as "Dependent Rational Animals", but let me precede that with a couple of comments for context: first, as I alluded in my last post referencing Levinas, it is my view that the the ethics demands a certain primacy in any healthy conception of life and society; second, in the area of ethics, Macintyre's After Virtue is the book that has had perhaps the biggest impact on my own thinking.

One of the criticisms of MacIntyre is that his critique of rational ethics is, on the one hand, devastating; on the other hand, his positive case for working out a defense of his own position - a revivification of social ethics in the Aristotelian-Thomist tradition(s) was somewhat pro forma. I think this is legitimate in so far as it relates to After Virtue itself (I believe I have read the latest edition - 3 - most recently), though I am not enough of a MacIntyre expert to offer a defensible critique of his work overall.

I do, however, want to draw attention to Dependent Rational Animals specifically in this light. Here MacIntyre begins with is the position of human as animal - as a kind of naturalist starting point for developing another pass at the importance of the tradition of the virtues. What is most remarkable is that in the process of exploring the implications of our "animality" MacIntyre manages to subvert yet another trajectory of twentieth century philosophy, this time as it relates to the primacy of linguistics. The net effect is to restore philosophical discourse back toward the reality of the human condition in the context of the broader evolutionary context of life on earth without - and this I must say is the most amazing part of this book - resorting to fables-masked-as-science (evolutionary psychology).

Pages