Re: Large XML Data, Parsing, JDBC, RMI, 2 Oracle DB

This sounds like a fun problem. (And something to put in a resume... reduced running time from 57 days to <insert anything way more reasonable here>) The fact that it's all in one file is kind of a pi9nt, that ruins delegating lists of small files off to different threads. I would have your main thread run a SAXParser (though this is probably obvious) and, since I don't know what your xml schema looks like, all I can say is, once you've parsed a portion which is some logical entity, pass that portion off to another thread to insert. Each thread might have some queue, filling up with these "logical portions". Ideally, you'd want as many "portion types" as threads, each thread would handle one type... this terminology is ugly, sorry. Did any of this make sense?

On Sat, 21 Jul 2001 at 12:04, Raden Java so generously added:

> Actually, the xml file is a one single huge file.
>
> In article <9jc4s2$4ri$2_at_cubacola.tninet.se>, "Klas Lindback"
> <klaslott_NO_SPAM_PLEASE_at_algonet.se> wrote:
>
> > Since the XML data is in several files you should be able to parse the
> > in parallel. The simplest form of parallellism is to launch several JVMs
> > and let each instance run just a few of the files. A little more
> > advanced form is to run your parser in several threads and let each
> > thread handle one file. You must make sure that the threads don't
> > interfere with each other. In your case you should make sure that they
> > won't deadlock each other in the db, or at least make sure the
> > transaction gets retried on deadlock. You must also check that your
> > classes are thread safe (look at the static methods and members).
> >
> > /Klas
> >
> >
> >
> >
>
 

-- 
Jason Simpson