Feed aggregator

• A support note that points to event 60025 to free up the stray LOB segments. 
• Tying the file upload/download component to one or two database sessions
• File size limits on uploads and downloads
• Is Apex the appropriate security gate-keeper to those fils
• Can we avoid dealing with the 'entire' file, and just have a few MBs at a time

Here’s a nifty video summary of a doctoral dissertation by Derek Muller that a client pointed out to me:

Click here to view the embedded video.

The basic gist is that students have pre-conceived notions that are wrong, and it is very hard to dislodge those mistaken notions. If you show them a video with an accurate explanation, the students will say that the video was clear and helpful, but they will misremember it as confirming their (mistaken) preconceived notions. In short, they won’t learn. In contrast, if you show them a video that starts by directly stating and then refuting their misconception, they like the video less and say it is confusing, but they actually learn more. This is a really important pedagogical point to know whether you are giving traditional in-class lectures, writing curricular materials, or creating one of those oh-so-modern video lectures that all the cool kids are into these days.

It’s also a good example of the kind of insight that big data is completely blind to. And it gives us good reason to be skeptical that taking large lecture courses online, turning them into REALLY large lecture courses (with nice videos), and expecting that new and more effective pedagogies will rise out of the data because, you know, science or something, is more of a hope (or a fantasy) than a plan to improve education.

Let’s say you have one of those ultra-hip MOOC platforms with a bazillion courses running on it and a hadoop thingamabob back end that’s tied to a flux capacitor, an oscillating overthruster, and a machine that goes “ping!” You’ve got all the big data toys. And let’s say that, among the many thousands of lecture videos being used on your platform, a bunch of them are designed the way Muller’s work suggests is best practice. Some of these were done this way consciously with awareness of the research. Some were done this way on purpose but based on intuitions by classroom teachers. They don’t have a name for what they’re doing, and they don’t really think about it as a general pedagogical strategy, but they have learned from experience that there are certain spots in their courses where they have to confront some misconceptions head-on. And then some of the videos may be in the Muller format completely accidentally. For example, maybe there’s a video of students working through a problem together. The first idea they come up with is the misconception, but they talk it through together and come up with the right answer in the end. This wasn’t planned, and the teacher who posts the video may not even be aware of why this sequence of events makes the event effective. Maybe she believes in the value of watching students work through the problem together and posts lots of student conversations videos, some of which end up being in Muller’s format and some of which don’t. Let’s assume that many of these videos are effective at teaching the concepts they are trying to teach, and let’s also assume that they are effective for the reason that Muller hypothesizes.

The first question is whether our super-duper, trans-warp-capable, dilithium crystal-powered big data cluster would even identify these videos as noteworthy. The answer is maybe, but probably not reliably so. Muller set up a controlled experiment with one variable designed to test a well-formed hypothesis. He was measuring whether this style video was more effective than the alternative of a more traditional lecture delivery. In science, this is called a “control of variables strategy.” In product development, it’s called “A/B testing” or “split testing.”

Big data usually doesn’t work that way. Instead of creating a tightly controlled set of conditions, it usually looks at what’s available “in the wild” and relies on the massive numbers of examples it has plus the power of computers to do lots of comparisons really fast to come up with inferences. Let’s say, for example, that you’re a medical researcher trying to figure out the role of genetics in a particular type of cancer. There are many, many genes that could be involved, and it may be that a bunch of them are involved but interact in complex ways. And, of course, environmental factors such as diet or exposure to carcinogens, as well as a certain amount of chance, can all impact whether a particular individual gets cancer. The good news is that, while there are many variables, they are finite in number, mostly known and measurable, and mostly have a quantifiable and reasonably regular impact on the cancer outcome (if you understand all the interactions sufficiently well). If you have a large enough database of patients with enough genetic material and good details on the non-genetic factors that you think probably contribute to the likelihood that they will get cancer, then a big data approach will probably help. There are regular patterns in the data. The main challenge is sifting through the mountains of data to find the patterns that are already there. Big data is good for that kind of problem.

But education doesn’t work that way. The same video may impact different students very differently, due to variables that mostly aren’t in our computer systems. For one thing, classes can be taught differently in many, many different ways, some of which matter and some of which don’t. Again, if we were doing a split test in a MOOC context, we could control the variables what happens when you just change one video for a class that is otherwise the same for many students. That approach has significant research value, but it’s not big data magic. It’s educators who come up with hypotheses and test them using a large data set. Students are also very different, in important ways that often don’t show up the data that we have in our online systems. Silicon Valley is not going to make us magically smarter about teaching.

Now, big data enthusiasts might argue that I’m not thinking big enough in terms of the data set, and that could make a difference. Knewton, for example, claims that their system can track students across courses and semesters and test hypotheses about them over time. For example, suppose a student is struggling with word problems in a math class. It’s possible that the student is having difficulty translating English into math variables, or trouble identifying the important variables in the first place. Those are both math-related issues. But it’s also possible that the student just has poor English decoding skills in general. Knewton claims that their system can hold all of these hypotheses about the student and then test them (presumably using some sort of Baysian analysis) across all the courses. If there is evidence in the English class that the student is struggling with basic reading, then that hypothesis gets elevated. And maybe that student gets extra reading lessons slipped in between math lessons. It sounds really cool. I haven’t seen evidence that it actually works yet, and to the degree that it does, it raises other questions about whether you need all student educational interactions to be on the platform in order to get the value, who owns the data, and so on. Put this one in the “maybe someday” category for now.

But even granting that you can get sufficiently rich information about the students, there’s another hard problem. Let’s say that, thanks to the upgrade in your big data infinite improbability drive made possible by your new Spacely’s space sprocket, your system is able to flag at least a critical mass of videos taught in the Mueller method as having a bigger educational impact on the students the average educational video by some measure you have identified. Would the machine be able to infer that these videos belong in a common category in terms of the reason for their effectiveness? Would it be able to figure out what Muller did? There are lots of reasons why a video might be more effective than average. And many of those ways are internal to the narrative structure of the video. The machine only knows things like the format of the video, the length, what kind of class it’s in, who the creator is, when it was made, and so on. Other than the external characteristics of the video file, it mostly knows what we tell it about the contents. It has no way for it to inspect the video and deduce that a particular presentation strategy is being used. We are nowhere close to having a machine that is smart enough to do what Muller did and identify a pattern in the narrative of the speaker. Now, if an educational researcher were to read Muller’s research, tag a critical mass of the relevant videos in the system as being in this style, and ask the machine to find other videos that might be similar, it’s possible that big data could help. It might come back with something like, “Here are some videos that seem to have roughly the same kind and size of effect on test scores as the ones with the Muller tag.” Maybe. Even then, you’d have to have human researchers go through the videos the computer flagged—and there might be a lot of them—to see which ones really use the same strategy and which ones don’t. That would be better than nothing, but it’s far from magic.

By the way, the low-tech method commonly used now is even worse. Not only is it useless, it’s actually harmful. A/B tests are rarely done on curricular materials, but surveys and focus groups where students self-report the effectiveness of the materials are common, particularly among textbook publishers. And in that situation, the videos that the students report to be harder and more confusing would actually be the more effective ones. But, lacking any measure other than the survey of their real effect on learning, the publishers (or teachers) generally would toss out the more effective videos in favor of the less effective ones.

Whether we’re talking about machine learning or human learning about how to improve education, the real problem is that we don’t have a vocabulary to talk about these teaching strategies, so we can’t formulate, test, and independently verify our hypotheses. In the machine learning example, we could create an arbitrary “Muller” tag in the system, but we don’t have a common language among teachers where we say “Oh, yeah, he’s using the confront-the-misconceptions (CTM) lecture strategy for that one. I prefer doing a predict-observe-explain (POE) experiment to accomplish the same thing.” If we had a widely adopted language that describes the details of why instructors think a particular aspect of their lecture or their discussion prompt or their experiment assignment is effective at teaching, then big data could be helpful because we could tag all our videos with pedagogical descriptions. We could make our theories about teaching and learning visible to the system in a way that it would be more able to test. And, perhaps even more importantly, human researchers could be more effective at collaborating with each other on testing theories of teaching and learning. Right now, what we’re trying to do is a little like trying to conduct physics research before somebody has invented calculus. You can do some things around the edges, but you can’t describe the really important hypotheses about causes and effects in learning situations with any precision. And if you can’t describe them with precision, then you can’t test them, and you certainly can’t get a machine to understand them.

More on this in a future post.

The post Why Big Data (Mostly) Can’t Help Improve Teaching appeared first on e-Literate.

I’ve hacked both the PHP and CSS that drive this website. But if I had to write PHP or CSS from scratch, I literally wouldn’t know how to begin.

Something similar, I suspect, is broadly true of “business analysts.” I don’t know how somebody can be a competent business analyst without being able to generate, read, and edit SQL. (Or some comparable language; e.g., there surely are business analysts who only know MDX.) I would hope they could write basic SELECT statements as well.

But does that mean business analysts are comfortable with the fancy-schmantzy extended SQL that the analytic platform vendors offer them? I would assume that many are but many others are not. And thus I advised such a vendor recently to offer sample code, and lots of it — dozens or hundreds of isolated SQL statements, each of which does a specific task.* A business analyst could reasonably be expected to edit any of those to point them his own actual databases, even though he can’t necessarily be expected to easily write such statements from scratch. 

*Actually, the vendor is Teradata Aster. After I showed them a draft of this post, they indicated that it’s OK to use their name in the post, and they fondly think they’re already doing what I suggest in their current product.

Similar thoughts apply to other software domains. If one of your selling points is some variant on “ease of development”, yet it’s difficult for you to supply generous amounts of sample code, then probably either:

• You’re really doing a great job at visual programming, point-and-click, or some other code-free paradigm. Congratulations!
• Or your product isn’t as easy to program for as you hope.
• Or you’re so confused as to what your product is used for that you can’t imagine what kinds of sample code to whip up.

Please note that these are not exclusive ORs.

I’m not suggesting “app stores where users can post and sell — or give away — their own apps”. Those may be good ideas (although probably not as good as you think), but they miss the point. You need to do the basic work yourself. Or, if it’s a big expensive deal for you to do the work, then you should make your product more usable. For if it’s hard for YOU to program in your technology, why would somebody else pay you so that they may have the privilege of doing so?

Recently I had to gather pending stats with Object Filter List by several objects, so I copied example from documentation and was surprised: instead of gather stats by specified filter list, oracle started to gather stats for all tables in current_schema! And ‘filter list’ applies only with dbms_stats.GATHER_DATABASE_STATS

UPD: Jared Still gave link to registered bug id, which i couldn’t find in MOS before:

Bug 12754926 – DBMS_STATS.gather_schema_stats does not work with an obj_filter_list parameter specified [ID 12754926.8]
Modified:04-Jan-2012 Type:PATCH Status:PUBLISHED
This issue is fixed in 11.2.0.4 (Future Patch Set)

SQL> exec dbms_stats.delete_schema_stats('HR');

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.31
SQL> begin
  2    for r in (select table_name
  3              from dba_tables t
  4              where t.owner='XTENDER'
  5              and table_name like 'TMP%'
  6    )loop
  7       dbms_stats.delete_table_stats('XTENDER',r.TABLE_NAME);
  8     end loop;
  9  end;
 10  /

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.31
SQL> select
  2    owner
  3   ,table_name
  4   ,num_rows
  5   ,last_analyzed
  6  from dba_tab_statistics t
  7  where owner='XTENDER' and table_name like 'TMP%'
  8        or owner='HR';

OWNER                          TABLE_NAME                       NUM_ROWS LAST_ANALYZED
------------------------------ ------------------------------ ---------- -------------------
HR                             COUNTRIES
HR                             DEPARTMENTS
HR                             EMPLOYEES
HR                             JOBS
HR                             JOB_HISTORY
HR                             LOCATIONS
HR                             REGIONS
XTENDER                        TMP
XTENDER                        TMP1
XTENDER                        TMP_CONTENT
XTENDER                        TMP_DATA
XTENDER                        TMP_DOCUMENT
XTENDER                        TMP_EXCH

13 rows selected.

Elapsed: 00:00:00.11
SQL> col dt new_value dt
SQL> select sysdate dt from dual;

DT
-------------------
2013-01-27 00:30:21

1 row selected.

Elapsed: 00:00:00.00
SQL> DECLARE
  2        filter_lst  DBMS_STATS.OBJECTTAB := DBMS_STATS.OBJECTTAB();
  3      BEGIN
  4        filter_lst.extend(2);
  5        filter_lst(1).ownname := 'XTENDER';
  6        filter_lst(1).objname := 'TMP%';
  7        filter_lst(2).ownname := 'HR';
  8        DBMS_STATS.GATHER_SCHEMA_STATS(NULL, obj_filter_list => filter_lst,
  9                                       options => 'gather');
 10      END;
 11  /

PL/SQL procedure successfully completed.

Elapsed: 00:03:16.89
SQL> select owner,table_name
  2  from dba_tables t
  3  where t.LAST_ANALYZED>timestamp'&dt'+0;

OWNER                          TABLE_NAME
------------------------------ ------------------------------
XTENDER                        TT
XTENDER                        TR_LOG
XTENDER                        IOT1
...[skipped 171 rows]...
XTENDER                        DEPARTMENTS

175 rows selected.

Elapsed: 00:00:01.04

This hCalendar event brought to you by the hCalendar Creator.

I'm increasingly amazed at IPython Notebook, and want to use it for an interactive tutorial. I'd like the notebook to be full of exercises for the student to fill out, and to get feedback from the notebook itself - a lot like CodingBat but in IPython Notebook format.

Here's the code: ipython_doctester

Next step - maybe - would be to finish with a scorecard cell summarizing the student's overall progress.

And then... automatically push data on a student's progress up to a webserver on the instructor's machine? Which presents a dashboard showing her where everybody is and what everybody is struggling with...

Like every open-source author, I'm eager for feedback!

This dip into IPython Notebook reminds me of people who suggested years ago that I should integrate SQLPython into IPy. Indeed, writing a SQL-handling extension for IPy sounds really attractive... I wonder how many insomniac nights it would take...

Happy Ada Lovelace Day!

Announcing the first

The Columbus Python Workshop for women and their friends is a free hands-on introduction to computer programming that's fun, accessible, and practical even to those who've never programmed at all before. We empower women of all ages and backgrounds to learn programming in a beginner-friendly environment.

Thanks to Pillar Technologies for hosting the workshop in their brand-new office in Columbus' Short North!

The workshop is the latest in a series based on the famous Boston Python Workshop; they've already introduced hundreds of beginners to programming in Boston, Indianapolis, Portland, Chicago, and Kansas City. Now it's Ohio's turn, so spread the word!

Get more details and sign up now: https://openhatch.org/wiki/Columbus_Python_Workshop

Traditionally, Ada Lovelace Day is celebrated by highlighting the achievements of present and past women in science and technology. Today, instead, I'm saluting the contributors of the future!

The big improvement for 0.2 is that ipython_doctester now optionally reports test results to an instructor's webapp, http://ipython-docent.appspot.com. In a classroom setting, this helps a teacher see who needs more time or help.

To turn on reporting, at the beginning of the IPython Notebook session,

import ipython_doctester
ipython_doctester.workshop_name = 'demo1'
ipython_doctester.student_name = 'Catherine'
from ipython_doctester import test

The instructor should run through the notebook first, executing all cells just to register each of the function names in order. Then she can check on the class's progress at http://ipython-docent.appspot.com/workshop_name.

You can also optionally set ipython_doctester.verbose = Trueto get the full table of results for every exercise, even the successes.

I plan to use this for the Columbus Python Workshop on Jan 18-19. Anyone you know from central Ohio needs to hear about this class!

That IPython project I was kicking around has started to mature, and I've released it into PyPI:

ipython_doctester

I'm much happier with using it as a decorator, it stays out of the student's way that way:

The first Columbus Python Workshop for women and their friends was a success!

We had about 18 students from all sorts of backgrounds, plus TAs Alyssa and Ryan from the Central Ohio Python User Group. They dived in with a will and learned an amazing amount in a short time. We all had fun and I loved having them there... there's something uniquely wonderful about teaching to faces that are smiling at you. By the end, we had several nice variants on the Color Wall.

Food sponsorship by LeadingEdje kept us all nicely fed and caffeinated, and Pillar provided The Forge, its new creative workspace in the Short North.

Over lunch on Saturday, COHPy members Ann Elliot and Thomas Winningham stopped in to show samples of some of the seriously fun things our Python programmers can look forward to as their skills increase. Ann showed some awesome astronomical charts she'd made with PyEphem, and Thomas used Python and an Arduino to make a string of multicolor LEDs do tricks. (With a little forethought, we could have married up the students' ColorWall scripts to Thomas's lights... maybe next time!)

For a first follow-up, some of us worked together at GirlDevelopIt Columbus's Hack Night:

... and we're expecting to see a bunch more at the next Central Ohio Python User Group meeting on Monday.

My experiment with running the exercises through IPython Doctester rather than CodingBat was... partially successful. Many students had to fall back on CodingBat thanks to trouble installing pyzmq on Windows. So I know some improvements to make for next time around.

To get involved with our next workshop in Columbus, sign up for the Central Ohio Python User Group. The next Ohio workshop, though, will be in Dayton... home sweet home at last!

If you came to the workshop, I would love for you to stay in touch, to ask questions or for any other reason. Thanks to all the students, the sponsors, and to the Boston Python Workshopfor leading the way!

Maybe the DELETE command should be banned from SQL. It was probably invented when the disks were tiny and it might not be suitable for our times, especially nowadays when we talk about things like BIG DATA.

Why do we have to delete anyway? We should invalidate and age out data and never destroy it. I am one of those who would say ‘keep it’ and never lose it. Even a typo, an error while entering somebody’s name in a text field, tells us a story. The number of attempts to get it right, the number of characters typed which were wrong, the time the wrong entry took place and much more, is data. Temporal databases come to mind. Why delete it?

But, one can argue that not deleting can be dangerous, too! One can claim that by refusing to delete we might end up with a lot of old, duplicated, bad quality, uncleaned, irrelevant and untrustworthy data.
 

Maybe the worse thing that can happen to data is to refuse to delete it enough. Perhaps because of this lack of enough deletes and fear of deletes we end up with all the 'bad data' found in some  database systems today.

Whatever you choose to do, delete or not delete, one thing's for sure and that is that you should know what you are deleting or not deleting. Choosing confidently what to delete comes from understanding and knowing your data model, and the implications a delete will have on the data model and never because the data is not required for your project! 

Some years ago I wrote about how sqlplus allows you to run sqlplus scripts directly from HTTP and FTP locations instead of the local filesystem. By the way, I didn’t even notice – my blog is over 5 years old already! :)

I a recent email thread Marco Gralike just showed the simplest way I to open a HTTP URL and download + list its contents in a CLOB datatype. It’s the HTTPURITYPE and its getCLOB (and getBLOB) methods.

So, basically if you want to download and save a (text) file like a script without a browser for some reason – and assuming that your database server can make outgoing HTTP connections, you can use this trick:

SQL> SET LINES 1000 TRIMSPOOL ON TRIMOUT ON PAGESIZE 0 LONG 99999999 LONGCHUNKSIZE 99999999 FEEDBACK OFF HEAD OFF
SQL> SPOOL $HOME/snapper_download.sql
SQL> 
SQL> select httpuritype('http://blog.tanelpoder.com/files/scripts/snapper.sql').getCLOB() from dual;
--------------------------------------------------------------------------------
--
-- File name:   snapper.sql
-- Purpose:     An easy to use Oracle session-level performance measurement tool
--              which does NOT require any database changes nor creation of any
--              database objects!
--
--              This is very useful for ad-hoc performance diagnosis in environments
--              with restrictive change management processes, where creating

... lots of output snipped ...

SQL> SPOOL OFF

Now you have to open the spool file (snapper_download.sql) and remove any garbage (sqlplus commands) from the top of the spool file and there you go:

SQL> @snapper_download all 5 1 3
Sampling SID 3 with interval 5 seconds, taking 1 snapshots...

-- Session Snapper v3.62 by Tanel Poder ( http://blog.tanelpoder.com )

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    SID, USERNAME  , TYPE, STATISTIC                                                 ,         DELTA, HDELTA/SEC,    %TIME, GRAPH       , NUM_WAITS,  WAITS/SEC,   AVERAGES
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      3, SYS       , STAT, opened cursors cumulative                                 ,            43,        8.6,         ,             ,          ,           ,
      3, SYS       , STAT, recursive calls                                           ,            44,        8.8,         ,             ,          ,           ,
      3, SYS       , STAT, recursive cpu usage                                       ,             9,        1.8,         ,             ,          ,           ,
      3, SYS       , STAT, CPU used by this session                                  ,             8,        1.6,         ,             ,          ,           ,
      3, SYS       , STAT, in call idle wait time                                    ,           412,       82.4,         ,             ,          ,           ,
      3, SYS       , STAT, session uga memory                                        ,         65512,      13.1k,         ,             ,          ,           ,
      3, SYS       , STAT, session pga memory                                        ,        327680,     65.54k,         ,             ,          ,           ,
      3, SYS       , STAT, calls to get snapshot scn: kcmgss                         ,            43,        8.6,         ,             ,          ,           ,
      3, SYS       , STAT, session cursor cache hits                                 ,            43,        8.6,         ,             ,          ,           ,
      3, SYS       , STAT, session cursor cache count                                ,             1,         .2,         ,             ,          ,           ,
      3, SYS       , STAT, workarea executions - optimal                             ,            48,        9.6,         ,             ,          ,           ,
      3, SYS       , STAT, execute count                                             ,            43,        8.6,         ,             ,          ,           ,
      3, SYS       , STAT, sorts (memory)                                            ,            45,          9,         ,             ,          ,           ,
      3, SYS       , STAT, sorts (rows)                                              ,          1867,      373.4,         ,             ,          ,           ,
      3, SYS       , TIME, PL/SQL execution elapsed time                             ,          5398,     1.08ms,      .1%, [          ],          ,           ,
      3, SYS       , TIME, DB CPU                                                    ,        435933,    87.19ms,     8.7%, [@         ],          ,           ,
      3, SYS       , TIME, sql execute elapsed time                                  ,        129913,    25.98ms,     2.6%, [#         ],          ,           ,
      3, SYS       , TIME, DB time                                                   ,        449166,    89.83ms,     9.0%, [#         ],          ,           ,
      3, SYS       , WAIT, PL/SQL lock timer                                         ,       4127427,   825.49ms,    82.5%, [WWWWWWWWW ],        41,        8.2,   100.67ms

--  End of Stats snap 1, end=2013-01-24 22:45:53, seconds=5

    

--  End of ASH snap 1, end=2013-01-24 22:45:53, seconds=5, samples_taken=41
 

Of course this technique is more for fun and is not reliable for binary files (unless you use something like UTL_ENCODE.BASE64_ENCODE or UUENCODE first). But still pretty fun :) I was happy to see that accessing a HTTP resource within the database has become so simple that it’s just a one-liner (as opposed to all the UTL_HTTP code lines needed for HTTP access).

Have fun (and thanks Marco for the tip :-)

• Sqlplus is my second home: Part 6 – Colorizing sqlplus
• Flexible Sqlplus command line history with RLWRAP
• MOATS-like sqlplus “top” utility for RAC
• Create a database link with the new host:port/service syntax
• Oracle Closed World presentation links