Re: Connection pool issues
Date: Tue, 17 Mar 2015 10:25:01 -0600
Message-ID: <CAJzM94D0E33LfRGM1oL4=yPp3c+diY6rT7aZp=AhCzDKPeGS4A_at_mail.gmail.com>
Final update. Turns out there was a code change that no one mentioned. The developer believed it wasn't a code change because it was new functionality in the application, so replied there was no code change when we asked. Yeah, right.
At any rate, once we knew about the change, it cleared up the mystery about the performance change. The new functionality resulted in a new SQL statement, which had never been run by the DBA team, optimized, or otherwise reviewed by anyone but the developer. We ended up creating a new index since we weren't allowed to change the query, which really could use some tuning. The result was that the new index alleviated the problems with the new query thereby resolving the connection issues since queries weren't taking over 24 hours to complete. In addition, the new index was picked up by 5 other queries and improved their performance as well. All 6 queries do a fast-full-scan on the index instead an FTS on a really large table.
Thanks everyone for your suggestions.
Sandy
On Thu, Mar 5, 2015 at 4:04 PM, Sandra Becker <sbecker6925_at_gmail.com> wrote:
> Unfortunately, the column most used in the predicates is the 4000 column.
> Got another DBA working on it right now so I can take a bit of a break.
> The developer did suggest a work around for the app support team that is
> helping them get their work done, just more slowly than the usual
> slowness. Some days I hate technology.
>
> Sandy
>
> On Thu, Mar 5, 2015 at 3:48 PM, MARK BRINSMEAD <mark.brinsmead_at_gmail.com>
> wrote:
>
>> 16K will probably do. The error you are getting concerns the size of the
>> *actual* data, not the declared datatypes. If your table has a bunch of
>> VARCHAR2(4000) columns defined, you will probably want the biggest blocks
>> you can get!
>>
>> (Note: once the index is created, you can *still* get errors on INSERT
>> or UPDATE if the new data exceeds the maximum size of an index key for the
>> block size chosen.)
>>
>> Omitting a column from the index will -- of course -- reduce its
>> effectiveness. I expect somebody has probably crafted an index that
>> includes ALL filter columns for the query, PLUS all columns in the
>> SELECT-LIST. If you find yourself stuck for space, you can remove some of
>> the columns in the SELECT list -- the cost of that will be (up to) one
>> additional IO for each row in the final result set.
>>
>> On Thu, Mar 5, 2015 at 5:39 PM, Sandra Becker <sbecker6925_at_gmail.com>
>> wrote:
>>
>>> That's what I was thinking. They want all the columns in the index,
>>> figures. We'll need a new tablespace with 16k blocks if I read everything
>>> correctly.
>>>
>>> Sandy
>>>
>>> On Thu, Mar 5, 2015 at 3:18 PM, MARK BRINSMEAD <mark.brinsmead_at_gmail.com
>>> > wrote:
>>>
>>>> You'll either need to drop some columns from the index, or create the
>>>> index in a tablespace with bigger blocks.
>>>>
>>>> The 3800 byte limit is a limitation of 8K block sizes.
>>>>
>>>> On Thu, Mar 5, 2015 at 5:16 PM, Sandra Becker <sbecker6925_at_gmail.com>
>>>> wrote:
>>>>
>>>>> We did notice that everything was slow, with the problem queries do
>>>>> the bulk of the IO. I think you're right about erecting barriers rather
>>>>> than solving the problem. After all, the database is going away. How does
>>>>> that help the customers now though? Sigh...
>>>>>
>>>>> So the analysis and testing show that a new index would improve
>>>>> performance for several queries. Problem is we can't seem to get it
>>>>> created in production. Keep hitting
>>>>>
>>>>> ORA-00604: error occurred at recursive SQL level 1
>>>>> ORA-01450: maximum key length (3800) exceeded
>>>>>
>>>>> Researching what we can do there now. The index columns add up to
>>>>> 4318. Another design flaw biting us.
>>>>>
>>>>> On Thu, Mar 5, 2015 at 2:28 PM, MARK BRINSMEAD <
>>>>> mark.brinsmead_at_gmail.com> wrote:
>>>>>
>>>>>> How "up to the minute" is the data, if the query takes more than 24
>>>>>> hours to run ?
>>>>>>
>>>>>> If you are *extremely* lucky, though, you can define a matview with
>>>>>> REFRESH FAST ON COMMIT. Then the data is ALWAYS fresh.
>>>>>>
>>>>>> On the other hand, if an mview that refreshes once per hour can
>>>>>> reduce runtime of the query from 24+ hours to less than a few minutes, it
>>>>>> has still made the final data returned about 23 hours "fresher". Oh! And
>>>>>> the data is updated by batch only once per day? That *is* stupid.
>>>>>>
>>>>>> It sounds rather like somebody is more interested in erecting
>>>>>> barriers than in solving the actual problem.
>>>>>>
>>>>>> As for IO speeds ... your *entire database* is suffering from
>>>>>> extremely slow IO, not just the "problem" queries. If you dig into the
>>>>>> details of the AWR report, though, I would not be shocked to find that your
>>>>>> "problem" queries account for the vast majority of PIO done by the database.
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 5, 2015 at 11:59 AM, Sandra Becker <sbecker6925_at_gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> When I started 18 months ago, I was told another database would be
>>>>>>> going away "soon". Looks like soon will be next year. Maybe. We just
>>>>>>> shutdown 2 databases yesterday that the product manager said they haven't
>>>>>>> used in 2 years. They forgot to mention they shutdown the applications.
>>>>>>> Hmmm...can you say "communication needs work"?
>>>>>>>
>>>>>>> We don't really have fast and slow queries doing the same IO, just
>>>>>>> that the queries they are not concerned about are expected to be slow.
>>>>>>> I've looked at them and the schema indexes and there is definitely
>>>>>>> opportunities for improvement if only the boss would let me.
>>>>>>>
>>>>>>> I thought of a materialized view, but they want up-to-the-minute
>>>>>>> data. Which is really stupid since the data is updated through batch only
>>>>>>> once per day. The big issue right now is physical IO. We're testing a new
>>>>>>> index right now that should reduce it considerably. It did on my test runs
>>>>>>> anyway. We'll have to see how the app behaves. When I get a chance, I
>>>>>>> also want to see if there is a way to tune the sql. Just too many fires
>>>>>>> for myself and the other oracle DBA this week. Not to mention a boss that
>>>>>>> really doesn't understand.
>>>>>>>
>>>>>>> oracle-l has been my salvation on many occasions.
>>>>>>>
>>>>>>> Sandy
>>>>>>>
>>>>>>> On Thu, Mar 5, 2015 at 8:18 AM, MARK BRINSMEAD <
>>>>>>> mark.brinsmead_at_gmail.com> wrote:
>>>>>>>
>>>>>>>> lol.
>>>>>>>>
>>>>>>>> I have had "databases that are going away soon" for as much as 5
>>>>>>>> years at a time. In fact, I "went away", and most of those databases are
>>>>>>>> probably still operating. :-)
>>>>>>>>
>>>>>>>> Its a little odd that you have "fast" and "slow" queries doing the
>>>>>>>> same amount of IO. That suggests pretty strongly that (just) IO is not the
>>>>>>>> problem. What are the WAIT events like on the slow running queries? Where
>>>>>>>> is the majority of the time going?
>>>>>>>>
>>>>>>>> It could be that your "problem" queries are doing something weird
>>>>>>>> like running HASH-JOINS inside a nested-loop, hashing the same darned data
>>>>>>>> over and over and over again. That would chew through CPU like nobody's
>>>>>>>> business, but do almost no (physical) IO. Logical IOs, though, would still
>>>>>>>> be through the roof. Are you making your tuning goal logical IO or
>>>>>>>> physical IO?
>>>>>>>>
>>>>>>>> We haven't even discussed the problem SQL. Might there be an
>>>>>>>> option to replace a complicated/expensive query with a materialized view?
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2015 at 10:04 AM, Sandra Becker <
>>>>>>>> sbecker6925_at_gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Yeah, I didn't understand the limit of 9 either, but the system
>>>>>>>>> was designed back when this was a 9i database and it's never been
>>>>>>>>> re-evaluated from what I've been told. In theory, the connection pool was
>>>>>>>>> supposed to support only a couple dozen users. They were discussing upping
>>>>>>>>> the pool to 20, but I haven't heard if that was done or not. They also
>>>>>>>>> have a second pool of 10 connections that had only 1 active session for a
>>>>>>>>> 24 hour period and recommended that our internal users hit that pool.
>>>>>>>>> Haven't heard back on that either. Been fighting yet another fire on
>>>>>>>>> another database.
>>>>>>>>>
>>>>>>>>> I did set up a cron job to automatically kill the sessions after a
>>>>>>>>> specific time limit based on information provided by the primary
>>>>>>>>> developer. Really smart guy who was involved with this app from the
>>>>>>>>> beginning. Hadn't thought of the resource limit. IOs could be a problem
>>>>>>>>> since some queries that are performing as expected do just as many IOs as
>>>>>>>>> the bad queries. Will have to give the CPU aspect some thought though.
>>>>>>>>>
>>>>>>>>> I also found that the queries in question are using the best index
>>>>>>>>> available, but it really isn't appropriate for the bulk of the queries that
>>>>>>>>> use it, including the bad queries. We are testing a more appropriate index
>>>>>>>>> in development now. Indexing on the entire schema needs to be reviewed,
>>>>>>>>> but I'm not being allowed to be proactive in addressing it since "the
>>>>>>>>> database is going away soon", soon meaning maybe September or 1st quarter
>>>>>>>>> next year or maybe 2nd quarter. Same old battle. Wait until it breaks
>>>>>>>>> before you do anything.
>>>>>>>>>
>>>>>>>>> Sandy
>>>>>>>>>
>>>>>>>>> On Wed, Mar 4, 2015 at 10:46 PM, MARK BRINSMEAD <
>>>>>>>>> mark.brinsmead_at_gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> A connection pool with only NINE connections? How unusual.
>>>>>>>>>> (Usually, when we feel the need to implement connection pooling, its
>>>>>>>>>> because we need to support nine-hundred or nine-thousand user sessions, not
>>>>>>>>>> nine.)
>>>>>>>>>>
>>>>>>>>>> But let's leave questions of architecture aside -- this is an
>>>>>>>>>> operational system, and I am happy to assume that it was designed this way
>>>>>>>>>> for a very good reason.
>>>>>>>>>>
>>>>>>>>>> It sounds like you have identified the cause of the problem.
>>>>>>>>>> Specific SQL statements running for excessive periods and holding the
>>>>>>>>>> connections until the (very limited) pool is exhausted.
>>>>>>>>>>
>>>>>>>>>> Until you have a more permanent fix, maybe you could consider a
>>>>>>>>>> work-around, like watching for and automatically killing any session that
>>>>>>>>>> has been executing the problematic SQL_ID for more than a reasonable time
>>>>>>>>>> (e.g., 2 hours).
>>>>>>>>>>
>>>>>>>>>> If you want to get really sophisticated, you can use resourse
>>>>>>>>>> manager to kill the long-running statement (as opposed to the session --
>>>>>>>>>> killing sessions may not be helpful, as this can "pollute" your connection
>>>>>>>>>> pool). The last time I looked into this -- probably on 10g -- I think I
>>>>>>>>>> found that resource manager could be used to limit CPU time, or IOs, but
>>>>>>>>>> not elapsed time. If I remember correctly, though, it did have the
>>>>>>>>>> interesting ability to refuse to *start* a statement if the
>>>>>>>>>> *predicted* runtime would exceed a threshold -- this might be
>>>>>>>>>> interesting here, although I would not be surprised to find that the
>>>>>>>>>> predictions will not be helpful enough.
>>>>>>>>>>
>>>>>>>>>> In the meantime, I have a hard time imagining how increasing the
>>>>>>>>>> size of your connection pool from 9 to 12 will starve the database server
>>>>>>>>>> of resources. 12 constantly-running, resource hungry sessions, on the
>>>>>>>>>> other hand, may be more of a problem. :-)
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 4, 2015 at 8:05 AM, Sandra Becker <
>>>>>>>>>> sbecker6925_at_gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Quick update. We completed creating the standby in the wee
>>>>>>>>>>> hours Sunday morning. Our connection pool problem is persisting. The
>>>>>>>>>>> connection pool allows only 9 connections, a number determined long before
>>>>>>>>>>> I came on board. I asked about the feasibility/impact of increasing the
>>>>>>>>>>> pool slightly. We decided it was best to keep it at its current level. We
>>>>>>>>>>> have identified a specific query, variations of the predicates, that can
>>>>>>>>>>> run more than 24 hours without finishing which, of course, means the
>>>>>>>>>>> connections are not released. According to the developer, this is a known
>>>>>>>>>>> problem with this database/query. His recommendation was to limit the
>>>>>>>>>>> search criteria and the number of people doing the search--this query is
>>>>>>>>>>> run by employees, not customers. In the meantime, sql analysis suggested a
>>>>>>>>>>> new index which I have put into the development environment for testing. I
>>>>>>>>>>> also will be looking at the query for additional optimization
>>>>>>>>>>> opportunities.
>>>>>>>>>>>
>>>>>>>>>>> Thank you all for your suggestions.
>>>>>>>>>>>
>>>>>>>>>>> Sandy
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 2, 2015 at 2:38 AM, Osborne, Chris <
>>>>>>>>>>> Chris.Osborne_at_bskyb.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, How big are you connection pools at the moment?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> They could be, counterintuitively, too big at the moment.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Chris
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Christopher Osborne*
>>>>>>>>>>>>
>>>>>>>>>>>> Lead Technical Specialist, Performance Engineering
>>>>>>>>>>>>
>>>>>>>>>>>> *British Sky Broadcasting*
>>>>>>>>>>>>
>>>>>>>>>>>> Email:chris.osborne_at_bskyb.com
>>>>>>>>>>>>
>>>>>>>>>>>> Desk: +44 1506 325069 | *Mobile: +44 7720 308941
>>>>>>>>>>>> <%2B44%207720%20308941>*
>>>>>>>>>>>>
>>>>>>>>>>>> *Please note new Mobile number. *
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [image: oebanner4ps_gap2_620]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *From:* oracle-l-bounce_at_freelists.org [mailto:
>>>>>>>>>>>> oracle-l-bounce_at_freelists.org] *On Behalf Of *Andrew Kerber
>>>>>>>>>>>> *Sent:* 27 February 2015 01:26
>>>>>>>>>>>> *To:* sbecker6925_at_gmail.com
>>>>>>>>>>>> *Cc:* MARK BRINSMEAD; oracle-l
>>>>>>>>>>>> *Subject:* Re: Connection pool issues
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It is extremely unlikely to be an issue with the standby unless
>>>>>>>>>>>> you are copying from active. If you are, you could be overloading network
>>>>>>>>>>>> and disk. Of course, that's just a shot from the hip with very little
>>>>>>>>>>>> information.
>>>>>>>>>>>>
>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Feb 26, 2015, at 2:51 PM, Sandra Becker <
>>>>>>>>>>>> sbecker6925_at_gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> We are discussing increasing the connection pool. Most
>>>>>>>>>>>> everyone is concerned that creating a standby from the active 19T database
>>>>>>>>>>>> is causing the application queries to run longer. Looking into it, but
>>>>>>>>>>>> this is our 5th try and no one reported any issues on the previous 4. The
>>>>>>>>>>>> recovery phase always failed. Still trying to figure that out, but that's
>>>>>>>>>>>> another discussion.
>>>>>>>>>>>>
>>>>>>>>>>>> Sandy
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 26, 2015 at 12:05 PM, MARK BRINSMEAD <
>>>>>>>>>>>> mark.brinsmead_at_gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> End users (and application developers) are almost always
>>>>>>>>>>>> "positive" that it is a database issue. :-)
>>>>>>>>>>>>
>>>>>>>>>>>> And it might be, too. Its just way too soon to know.
>>>>>>>>>>>>
>>>>>>>>>>>> In the meantime, if the database server is not busting at the
>>>>>>>>>>>> seams, you might want to consider configuring your app servers to allow the
>>>>>>>>>>>> connection pools to grow a little larger. The REAL error is that the app
>>>>>>>>>>>> server is not allowing the app to open a new connection.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 26, 2015 at 1:43 PM, Sandra Becker <
>>>>>>>>>>>> sbecker6925_at_gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you. Definitely some things I can look at. I was told
>>>>>>>>>>>> there were no changes to the code or app servers. Of course, the end user
>>>>>>>>>>>> is positive it's a database issue. I can definitely do AWR before/after.
>>>>>>>>>>>> We did have some network blips earlier in the week, but nothing today.
>>>>>>>>>>>>
>>>>>>>>>>>> Sandy
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 26, 2015 at 11:39 AM, MARK BRINSMEAD <
>>>>>>>>>>>> mark.brinsmead_at_gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> It sounds to me like you should be as interested (or
>>>>>>>>>>>> more) in connected sessions (as reported by gv$session).
>>>>>>>>>>>>
>>>>>>>>>>>> There is no need for *anything* to have changed in the
>>>>>>>>>>>> database -- it is entirely possible that a change in the application code,
>>>>>>>>>>>> or even in the application server can cause the problems you are
>>>>>>>>>>>> encountering.
>>>>>>>>>>>>
>>>>>>>>>>>> Possible issues would be:
>>>>>>>>>>>>
>>>>>>>>>>>> (*) Queries running longer than they used to, meaning that
>>>>>>>>>>>> applications need to hold connections from the pool open longer. There is
>>>>>>>>>>>> no reason to expect that these would necessarily appear as "longops",
>>>>>>>>>>>> though.
>>>>>>>>>>>>
>>>>>>>>>>>> (*) Application threads not releasing connections when (as soon
>>>>>>>>>>>> as) they are done with them.
>>>>>>>>>>>>
>>>>>>>>>>>> (*) More application threads running that before, perhaps due
>>>>>>>>>>>> to increased user population or other factors.
>>>>>>>>>>>>
>>>>>>>>>>>> (*) JDBC connection pool management, causing the maximum number
>>>>>>>>>>>> of available connections to be fewer, or causing "idle" connections to be
>>>>>>>>>>>> returned to the pool more slowly.
>>>>>>>>>>>>
>>>>>>>>>>>> (*) Changes to limits on session-idle-time, forcing idle
>>>>>>>>>>>> sessions to disconnect from the database.
>>>>>>>>>>>>
>>>>>>>>>>>> I bet I could think of at least 5 more causes if I thought
>>>>>>>>>>>> about it for another 10 minutes. I haven't even considered networking
>>>>>>>>>>>> problems yet!
>>>>>>>>>>>>
>>>>>>>>>>>> Rebuilding indexes would cause execution plans to be
>>>>>>>>>>>> invalidated, and might cause new plans to be chosen. It is possible that
>>>>>>>>>>>> you have seen an adverse change in plans on a commonly run query. (Check
>>>>>>>>>>>> AWR and Statspack for top queries by elapsed time, before and after the
>>>>>>>>>>>> "event".) But its just as possible that the problem has nothing to do with
>>>>>>>>>>>> the database at all.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 26, 2015 at 12:30 PM, Sandra Becker <
>>>>>>>>>>>> sbecker6925_at_gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Oracle - EE 11.2.0.2
>>>>>>>>>>>>
>>>>>>>>>>>> This week users started having issues connecting to a
>>>>>>>>>>>> database. Error below:
>>>>>>>>>>>>
>>>>>>>>>>>> Could not retrieve documents from the docStore: exception was
>>>>>>>>>>>> com.ghx.docstore.DocStoreException: Cannot get Connection to: db_arch
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In the log someone also saw
>>>>>>>>>>>>
>>>>>>>>>>>> Caused by: oracle.ucp.UniversalConnectionPoolException: All
>>>>>>>>>>>> connections in the Universal Connection Pool are in use
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Obviously, something isn't releasing the connection back to the
>>>>>>>>>>>> pool. I've queried longops to no avail. I've seen sessions using a
>>>>>>>>>>>> tremendous amount of CPU or I/O. When those sessions were killed, we see
>>>>>>>>>>>> an immediate uptick in processing. The only change to the database
>>>>>>>>>>>> occurred last Friday--we removed the oldest year of data from partitioned
>>>>>>>>>>>> tables and rebuilt unusable indexes. No code releases were done.
>>>>>>>>>>>>
>>>>>>>>>>>> My question: Is there anything I can query to see what
>>>>>>>>>>>> session(s) are causing the problem other than what I'm already doing?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions are appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Sandy
>>>>>>>>>>>> GHX
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Sandy
>>>>>>>>>>>> GHX
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Sandy
>>>>>>>>>>>> GHX
>>>>>>>>>>>>
>>>>>>>>>>>> Information in this email including any attachments may be
>>>>>>>>>>>> privileged, confidential and is intended exclusively for the addressee. The
>>>>>>>>>>>> views expressed may not be official policy, but the personal views of the
>>>>>>>>>>>> originator. If you have received it in error, please notify the sender by
>>>>>>>>>>>> return e-mail and delete it from your system. You should not reproduce,
>>>>>>>>>>>> distribute, store, retransmit, use or disclose its contents to anyone.
>>>>>>>>>>>> Please note we reserve the right to monitor all e-mail communication
>>>>>>>>>>>> through our internal and external networks. SKY and the SKY marks are
>>>>>>>>>>>> trademarks of Sky plc and Sky International AG and are used under licence.
>>>>>>>>>>>> Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited
>>>>>>>>>>>> (Registration No. 2067075) and Sky Subscribers Services Limited
>>>>>>>>>>>> (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc
>>>>>>>>>>>> (Registration No. 2247735). All of the companies mentioned in this
>>>>>>>>>>>> paragraph are incorporated in England and Wales and share the same
>>>>>>>>>>>> registered office at Grant Way, Isleworth, Middlesex TW7 5QD.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Sandy
>>>>>>>>>>> GHX
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Sandy
>>>>>>>>> GHX
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sandy
>>>>>>> GHX
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sandy
>>>>> GHX
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Sandy
>>> GHX
>>>
>>
>>
>
>
> --
> Sandy
> GHX
>
-- Sandy GHXReceived on Tue Mar 17 2015 - 17:25:01 CET
-- http://www.freelists.org/webpage/oracle-l