Michael Feldstein

Michael and I will be speaking at and attending a few events coming up in April.

ELI Online Spring Focus Session - Michael and I will jointly kick off the EDUCAUSE Learning Initiative (ELI) Online Spring Focus Session “Learning and the MOOC” on April 3 – 4. Our presentation is April 3 from 12:15 – 12:50pm ET:

Everything You Know About MOOCs Could Be Wrong  There are four things that we think we know about MOOCs: They are “massive,” they are “open,” they are “online,” and they are “courses.” But what happens if we start playing with those “truths”? How massive do MOOCs need to be? How open? Do they need to be fully online? Do they even need to be courses? As institutions determine whether and how to incorporate new online models into their repertoire, Phil Hill and Michael Feldstein will explore the boundaries of the current definition of the MOOC and discuss how it is likely to evolve in the next 12-24 months.

In addition, we will be helping Veronica Diaz and Malcolm Brown with the closing session, April 4 from 4:40 – 5:00pm ET.

Pearson Cite - I will physically be at the Pearson Cite conference, also April 3 – 4, and will present during the closing session April 4 from 4:45 – 5:45pm CT.

Online Educational Delivery Models: What to Expect Next For almost two decades, the world of online education has been evolving. Entering 2013, we are presented with a confusing array of educational delivery models. Ranging from relatively traditional classrooms to fully-online courses, online education is being integrated into the college experience for almost one-third of students. Many of these models, such as Massive Open Online Classes (MOOCs), are geared towards accessibility and affordability. Other models focus on personalized learning styles, such as competency-based education. Traditional LMS providers and textbook publishers are also trying to find their places in this new world, promoting yet different models.

American Association of Colleges of Nursing - I will be the keynote speaker for the AACN Hot Issues conference April 11 from 1:30 – 3:00pm CT.

Online Educational Delivery Models: A Descriptive View Consider varied approaches to online courses and programs, including fully online programs, School-as-a-Service, educational partnerships, competency-based education, blended/hybrid courses, the flipped classroom, and Massive Open Online Courses (MOOCs).

Education Innovation Summit - Michael will attend the GSV Advisors / ASU Education Innovation Summit April 15 – 17.

The post Upcoming Events for Phil Hill and Michael Feldstein appeared first on e-Literate.

Last week Steve Kolowich wrote in the Chronicle of Higher Education about the State University of New York (SUNY) system approval of a plan that would target reduced time-to-degree.

The State University of New York’s Board of Trustees on Tuesday endorsed an ambitious vision for how SUNY might use prior-learning assessment, competency-based programs, and massive open online courses to help students finish their degrees in less time, for less money.

The plan calls for “new and expanded online programs” that “include options for time-shortened degree completion.” In particular, the board proposed a huge expansion the prior-learning assessment programs offered by SUNY’s Empire State College.

Although the MOOC angle led the article’s title and online discussion, there was more significant news in terms of structural change regarding Prior Learning Assessments (PLA).

Even before the SUNY announcement, it had already been a big week for nontraditional models for awarding college credit. The U.S. Education Department on Monday said it had no problem with spending federal student aid on college programs that give credit based on “competency,” not the number of hours students spend in class.

Empire State College’s prior-learning assessment programs operate on a similar principle. Students who can demonstrate that they have acquired certain skills can get college credit, even if they did not acquire those skills in a college classroom.

The new SUNY effort will aim to copy the Empire State model across the system, said Nancy L. Zimpher, the chancellor.

“This resolution opens the door to assurances to our students that this kind of prior-learning assessment will be available eventually on all our campuses,” said Ms. Zimpher in an interview.

Background on PLA

Prior Learning Assessment, or PLA, is a little-discussed strategy to facilitate time-to-degree, particularly for non-traditional students. The concept is to set up the structure and processes to evaluate corporate training from employment, military training, civic responsibilities, travel, and independent study and award academic credit from these out-of-the-classroom learning situations. As the higher education population diversifies with much higher percentages of working adults, PLA can be an important factor in reducing total cost and time-to-degree.

In 2010 the Council For Adult & Experiential Learning (CAEL) published a study that was funded by the Lumina Foundation. Some of the key findings:

The data from 62,475 students at the 48 postsecondary institutions in our study show that PLA students had better academic outcomes, particularly in terms of graduation rates and persistence, than other adult students. Many PLA students also shortened the time required to earn a degree, depending on the number of PLA credits earned.

 

Paul Fain wrote last May at Inside Higher Ed about the quiet, but growing, role that PLA is taking within higher education policy circles.

But prior learning assessment mostly occurs behind the scenes, partially because colleges avoid loudly advertising that they believe college-level learning can occur before a student ever interacts with faculty members.

That low profile is ending, however, as prior learning is poised to break into the mainstream in a big way. The national college completion push and the expanding adult student market are driving the growth. And ramping up to meet this demand are two of the field’s early adopters — the Council for Adult and Experiential Learning and the American Council of Education — which may soon be even bigger players in determining what counts for college credit.

ACE is taking a lead role in trying to ensure quality standards in how PLA credits are awarded, and CAEL is playing a complementary role. Again from Fain’s article:

The association has decades of experience on prior learning assessment, as well as a large network of faculty credit evaluators and, perhaps most importantly, clout with its 1,600 college and university members.

“Who better than ACE to map out an adult learning agenda?” says Bataille.

Observers say prior learning’s impact on higher education could be enormous. Its potential could even rival that of online learning, by continuing to open student access beyond the campus, although the practice gets far less attention than splashy ventures like Sal Khan’s massively open online courses (MOOCs).

“Prior learning is the next phase,” says Ed Klonoski, president of Charter Oak State College, an online, public institution in Connecticut that conducts prior learning assessments. “It’s the next disruption.”

Loosely defined, there are four primary methods of assessing learning outside the classroom: through student portfolios; ACE credit recommendations based on corporate or military training programs; reviews conducted by individual colleges; and exams used to verify “learning achievements.” Those exams include the College Level Examination Program (CLEP), Excelsior College Exams and the DANTES Subject Standardized Tests.

Prior Learning Assessments and Competency-Based Education (CBE) – Brothers in Arms

Both PLA and CBE are based on the notion of moving beyond using seat time as the foundation of college credit, and both are biased towards non-traditional working adults. That’s why last week’s clarification from the Department of Education – that colleges can award financial aid based on “competencies” and not just seat time – is significant to both models. In Kelly Field’s article for the Chronicle, there was mention of this connection for PLAs.

Amy Laitinen, deputy director for higher education at the New America Foundation, said she hopes the department will expand its direct-assessment authority to remedial education and test the idea of awarding aid for prior-learning assessments.

“This letter really opens the doors to other things,” she said. “They are showing an interest in collaborating, in making this an ongoing conversation.”

Empire State College

Empire State College, a campus within SUNY, was one of the first colleges in the US to implement PLA and has provided credits in the third method mentioned above – “reviews conducted by individual colleges” – since the 1970s. Last year the Lumina Foundation awarded Empire State College a $500,000 grant to develop SUNY REAL (for Recognition of Experiential and Academic Learning) as a pathway for degree completion using open educational resources (OERs).

SUNY REAL, a two-year, $500,000 project, has the potential to expand capacity and access to higher education, increase the number of college graduates, decrease time to completion, reduce costs and assure quality.

The project will be a scaling up of the college’s nationally recognized programs for assessing college-level learning acquired outside of the formal higher education system; specifically, the project seeks to design, develop, deploy and disseminate a methodology to award credit for OER courses. Facilitating credit transfer and degree completion also are important aspects of the project. [snip]

Designed to be scalable and to serve all 64 SUNY campuses, SUNY REAL focuses on four key areas:

• assessment structures to evaluate verifiable, college-level, prior experiential and emergent learning for college credit
• recruitment and training of faculty and equivalent field experts for assessments
• transcription of the approved college-level learning
• ongoing research on its practices to ensure quality and consistency, and a link to the wider community of practice of noncollegiate learning assessment.

For more information Empire State College’s expanding role in open education, see this EDUCAUSE article.

Back to the SUNY Announcement

And this brings us back to last week’s news, in that SUNY has approved the expansion of SUNY REAL to cover all 64 campuses and potential audience of almost a half million students. What we now have is a project that would enable PLA-based credit for any SUNY student. From one of the press releases:

“We also are helping our students to translate real-world experience – such as military service or time in the workplace – into college credit,” said Zimpher. “Our program, which is under development at Empire State College, is called SUNY REAL. It will meet the learner where he or she is, and through prior-learning assessment by highly qualified faculty experts, will dramatically diminish the need for the repeat courses or costly skills training they already have. It is being designed for scale-up throughout SUNY.”

The SUNY effort, once implemented, would qualify it as the biggest implementation of PLA in the country, and I suspect it will raise the level of awareness around this important strategy. Consider also that in a recent survey of college and university presidents conducted by Gallup for Inside Higher Ed, there was strong support for the potential of competency-based education and Prior-learning assessments, which is another indication that PLA’s time has come.

There is fairly widespread enthusiasm about awarding academic credit based on students’ competency rather than seat time, with 21 percent of presidents strongly agreeing and another 39 percent agreeing that awarding competency- based credits has “great potential” for higher education. Support was stronger among public (68percent) and for-profit college presidents (66 percent) than among those in private nonprofit higher ed (49 percent).

Public sector presidents (26 percent) were more likely than private sector leaders (15 percent) to strongly agree that the use of prior learning assessments has potential for positive impact, and significantly larger numbers of presidents in most categories said they agree.

I expect we’ll see much more about PLA in the next few years.

Update: Added link to Empire State College article.

The post SUNY and the Expansion of Prior Learning Assessments appeared first on e-Literate.

There’s something that drives me a little crazy when I hear about how someone has learned from Netflix, Amazon, Google, etc., and that they’re going to be the Netflix, Amazon, or Google of education.  Actually there are a few things.

There is a natural tendency to want to leverage the work of others.  In the burgeoning space of learning analytics many look to those who have large data sets and algorithms to extract some kind of meaning from those massive stores of information.  We should always be looking to leverage when we can!  This time, though, our takeaways are limited.

What are Our Outcomes?

Amazon, Netflix, etc., have an outcome.  Let’s call it “profit” for lack of a better term.  They run data analysis to provide suggestions like Netflix does to recommend movies to users.  Why? The goal is to show you something you didn’t think of that you’d like, which then increases interaction and loyalty, leading to profit.  So long as they get that right even 1 in 100 times, it’s worth their while to run their analysis and make the suggestion.  For someone doing data analysis, this is a gift.  Any data mining algorithm wants something to optimize for.  Let’s say in this case, it’s “$”.

They can ask their algorithm “I did X, did they give us more $?”  Netflix can ask their data “I did Y, did I get more $?”  Google can ask “I did Z, did I get more $?”  Not only are they measuring the outcomes directly (more $) but also the inputs (X, Y, Z – whatever changes they introduce to their products).

We don’t have it that easy in education if we want to do it right (assuming we’re not looking just for more $).  Measuring learning is hard.  As Michael Feldstein discusses in his post A Taxonomy of Adaptive Analytics Strategies, “Since doing good learning analytics is hard, we often do easy learning analytics and pretend that they are good instead.”

I’ll be the first to say there is good work going on in early warning systems based on click data.  The outcome is yes/no – does the student stay?  They’re not measuring if students understand the material, just “do they stick it through.”  Don’t get me wrong – every student kept in school via an early warning system is an achievement.  But it’s not the big promise of learning analytics.  It’s an achievement of website analytics very similar to those studied by the commerce sites.  The more a user stays engaged with their sites, the more profit they generate.  The comparisons to those kinds of analytics pretty much end there.  Unfortunately for those looking for the easy path, our outcomes are complex and the inputs aren’t actually that obvious either.  Let’s talk about the real promise of learning analytics.

We need to be measuring learning the entire time in which students are engaging in learning activities and track how we believe those activities should be shaping the experience.

Explanatory vs. Predictive

How does a good teacher know if a student is struggling with a concept in the classroom?  We hope that they recognize signs of difficulty while reviewing practice work or are asked for assistance by the student (feedback, hints). If learning analytics are going to provide useful feedback then we should be measuring those feedbacks and requests for help.  A click stream tells me if a student is using material but not why or what that interaction ought to achieve.  A student might skip problems they already know – their lack of answering questions in a particular part of the course is not itself evidence of lack of understanding.  Similarly, a student can struggle while working very hard to try and understand a concept.  Their mere frequency of interaction does not in any way imply instructional success.  Only knowing their clicks, or visits, tells us nothing about their intent, when they wanted help, if they got that help, or what feedback they were given (or should have been given).  Consider the following questions one might want to ask:

• How often is the student getting questions right on the first try?
• Do they eventually get them correct?
• How often are they asking for help?
• Do expert teachers rate this skill is generally difficult?

Answering these (and many more such) questions require semantic data that we need to be collecting and cannot collect with a mere click-stream.  When Feldstein refers to Semantic Analytics, this points the finger at the algorithms.  It is also the lack of semantic data for algorithms to take advantage of.  What does the difference in that data look like?  This is an example of what I mean:

• Click-stream:
  • Student x clicked y at time z

• With semantic data, we can can store:
  • Student x has asked for his second hint on part three of the question “What are the five steps of this program?” and he was told “Recall that you need to identify a base case for your function” The correct answer will be “line 5”.  The question is related to the skill “recursive base case” and is often mistakenly answered as “line 4” due to a common misconception

• Click-steam:
  • Student x’ clicked y’ at time z’
• With semantic data we can store:
  • Student x’ has now selected “line 5” which is correct.  Student was given the feedback “You are correct, line 5 is the base case”.   This was her third try, though it is the first question about the skill “recursive base case” she attempted to answer even though it’s the third related question in the material.

Not only do we need data about interaction, but we need the content itself to identify these in meaningful ways.  If nobody tells the system what the hints and feedback mean, what skills the targeted interaction is meant to address, etc., the algorithms can’t make any reasonable estimations of learning.  It’s beyond the capacity of a simple algorithm to identify what these clicks mean without guidance of better data. We can go further.

• When do students ask for a specific hint and what do we know about the misconception they’re exhibiting?
• Does one hint for a  particular question provide enough guidance or are more hints needed?
• After selecting an answer and being told “That’s not quite right, because…” do they then answer correctly?

To answer these kinds of questions you have to have a design process that not only creates these targeted hints and feedback, but allows the system to semantically record each and every selection of students interacting with the material as described.  Only then can you ask the questions above of the data you’re collecting with any hope of a meaningful result.  If there are no targeted hints that students can ask for, if there is no targeted feedback, if there is no well-designed question, there is no semantic data.

What can we do when we are empowered with this sort of semantic data and analysis?  Here are just some examples:

• Provide real-time feedback to teachers about how groups of students and individuals are performing while they learn before summative exams or projects arise
• Provide guidance on where specifically in the course students are struggling
• Use semantic analyses like learning curve analysis to identify areas where content needs to be improved

This last one in particular is important.  A lot of times we are tempted to assume that whatever content we create simply works.  This is true even in the traditional classroom where a teacher prepares and delivers a lecture.  Was it an effective lecture?  We might be able to decide if students liked the lecture but we have the same data collection problem as the click stream.   All we can reliable know is if the student received the lecture.  In reality we want to be able to know in what areas are we being successful at imparting knowledge and be accepting of the fact that not everything is great right at the start.  Without semantic data and algorithms you’re forced to assume effective content and that when the student receives it, it did what it was supposed to (whatever that was) effectively (whatever that means).  With so many variables, we must make assertions that can then be verified or debunked by analyses.

It’s tempting to try to work around this metric problem by using a summative evaluation as the metric, ie, if they pass this test, then clearly all the stuff before must have done what we wanted.  (And even then, if they don’t, there’s no good information on why they didn’t).  This is not much better than saying the SAT is an accurate reflection of an individual’s skills as a whole and their educational experience up to exam time.  We want an approach that utilizes the formative work of students to give us insights.  If our materials are working, then the formative ought to simply give the same results and the summative ought to be perfunctory.

Impact of Errors in Analytics

The ultimate goal of learning analytics must be providing actionable feedback that can be given to students and instructors during the run of a course.  This is already possible technologically and pedagogically.  As this methodology becomes more readily available it will become expected.  This isn’t a pipe dream.  This mandates moving beyond the goal-post metrics of summative exams and click-stream data that allows us at best only look back and say “we weren’t entirely successful in supporting students last semester, so we’ll take a guess this semester at improving things for students next semester and see what happens.”  We can and ought to do better.

But this has to be done with a high degree of accuracy.  If Amazon gives me a recommendation I don’t agree with (which it actually does fairly often for me), is there any real harm?  If I buy some gifts for a friend and Amazon uses that to suggest other purchases for me that I don’t want, I don’t leave Amazon because I see a poor recommendation.  When they make a particularly good one their profit metric goes up a tic.  No harm, no foul and so it is worth Amazon making the occasional poor recommendation to capitalize on the good ones.

What if we’re not good at these predictive models?  It’s not as neat and tidy as recommendations.  If we mistakenly identify a student as at risk for dropping out, the two possible negative effects are that we intervene with someone who is not actually at risk (not a terribly bad thing) or we miss an at risk student we would otherwise hope to identify (not optimal but no different from if we weren’t trying).

Now what happens if we tell a student they aren’t achieving learning outcomes when in fact we are wrong about that?  The potential for demotivating the student comes at a high cost.  This could happen with errors in reporting the other way, as well.  If learning analytics inform a student they are succeeding but in fact they are not prepared for their next exam or job, the disservice is just as bad.  Getting learning analytics wrong on the learning dimension is a recipe for disaster and must be done carefully and with understanding.  Without that semantic ability to understand what is happening, we won’t even know if we’re doing harm to our students by using algorithms to optimize for things we don’t understand.

Summary

The way to take advantage of online learning technologies has to include the ability for timely, reliable, prescriptive information for those engaged in the learning experience (students and their instructors) as well as a rich semantic data set for learning engineers to be able to improve those resources continually.  The only way to do this is to have algorithms and data that have an explanatory capability in order to give guidance to each user group on what to do next.  This means developing rich models that encapsulate the intent of online learning content and well-instrumented learning environments that provide large sets of meaningful data that can feed these analyses.  Click streams provide retention data and this is being used to success.  Now we need to recognize the next step is in fact a harder one to take, but well worth it for everyone involved.

 

The post If you like learning, could I recommend analytics? appeared first on e-Literate.

I’m pleased to announce that Bill Jerome, the Associate Director of User Experience at OLI, will be writing some guest post on educational data analytics and related topics. This is an area where there is a lot of industry hype and not a lot of understanding right now. I’m really pleased to have an expert like Bill here to help us gain some insight.

The post Please Welcome Guest Blogger Bill Jerome appeared first on e-Literate.

A couple of weeks ago in my post about the different types of learning analytics, I described retention early warning systems thusly:

Most people don’t think about early warning systems as being in the same category as adaptive analytics, but if you consider that “adaptive” really just means “adjusting to your personal needs,” then a system like Purdue’s Course Signals is, in fact, adaptive. It sees when a student is in danger of failing or dropping out and sends increasingly urgent and specific suggestions to that student. It does that without “knowing” anything about the content that the student is learning. Rather, it’s looking at things like recency of course login (Are you showing up for class?), discussion board posts (Are you participating in class?), on-time assignment delivery (Are you turning in your work?), and grade book scores (Is your work passing?), as well as longitudinal information that might indicate whether a student is at-risk coming into the class. What Purdue has found is that such a system can teach students metacognitive awareness of their progress and productive help-seeking behavior. It won’t help them learn the content better, but it will help them develop better learning skills.

Well, last week, Ray Henderson announced Blackboard’s new Retention Center and described it as follows:

The Retention Center gives critical insight on learning and activity gaps to instructors, within the LMS, that helps them quickly diagnose students that are falling behind. Pre-configured and automatic so they don’t have to hunt for it. No set-up: it automatically calls out students that are at risk while instructors still have time and space to do something about it. With the feature instructors can see:

• Who’s logging in: this is a simple but powerful predictor of student success. Instructors see how long it’s been since students have logged in to the course and how many have been away for five days or more. And not by fishing through student profiles or reports but in an automatic view complete with red flags where they’re needed.
• Whether they’re engaged: which students have had low levels of course activity, at 20 percent or below the average in the last week.
• Whose grades are suffering: which students are currently trending at 25 percent or more below the course average so they can target extra help to where it’s most needed – even when it isn’t asked for.
• Who has missed deadlines: instructors might know this anecdotally or on a case-by-case basis, but now they can get a real-time view of all students that have missed one or more deadline.

Eerily similar, no? A number of years back, when I pressed Course Signals inventor John Campbell on which factors in the LMS are most highly predictive of student success across different courses, he named exactly these four. The only surprise here is that this isn’t a common analytics feature of every LMS and courseware platform on the market yet. Purdue proved that their value in helping at-risk students is high. I’m glad Blackboard is stepping up.

The one piece that’s missing is a simple standard where an SIS or other longitudinal data system could pass an at-risk “credit score” to the early warning system to modify its sensitivity. If a student on the honor roll drops off the radar for a week, it’s less of a cause for concern that a student on academic probation (for example). I tried to push this idea for a standard at the IMS a few years back but got nowhere with it at the time. I hope that Blackboard will push for something like it now that they have a system to take the data.

The post Blackboard’s New Early Warning Analytics Product appeared first on e-Literate.

I previously shared the text of SB 520, the proposed California legislation that would identify and approve a set of up to 50 online courses that the three public systems would accept as credit for admitted students. In my notes for the press conference introducing the bill, there are updated links to most major press articles on the bill as select blog posts. Michael shared his analysis  of the bill and where he thinks changes are needed.

The more time I have to think about this news, the more I’m convinced that if successful, the passage of this bill (or an amended version substantially meeting the stated aims described in the press conference) could have an impact much bigger than California students taking online courses. This bill aims to establish a new right – for admitted students to have access to the courses they need.

The right for admitted students to have courses available

The real significance of SB 520 is that it focuses on the student, not the institution, and specifically on admitted students. When the Master Plan was adopted in California starting in 1960, the basic premise was to guarantee students a place within one of the three public systems based on their high school record. It was assumed that by having a place in a public institution, the student would have access to needed courses.

For various reasons, this assumption is no longer valid.

As Hillary Hill described at e-Literate (yes, she’s my daughter) and as Rich Copenhagen described at the SB 520 press conference, there is a crisis for enrolled students in trying to get into the courses they need. What good is being admitted if you still can’t complete your education?

Rather than directly address the institutions and how they operate, SB 520 focuses on the student and (if successful), this approach will change the conversation. Admitted students would have the right to get the lower-division courses they need, and if the school cannot provide the courses, there will be a release valve of online courses that the schools have to accept for credit.

Changing risk / reward, but letting colleges decide

This approach, while not directly addressing what any individual college or university should do, does change the risk / reward structure. There is a strong argument that institutions will fight this bill for the reason that high-enrollment lower-division courses are in fact the biggest money-makers for a school. By the availability of these courses from online providers, schools will now have greater motivation to provide the courses for more students who need them, if the schools want to keep the revenue.

If a school chooses to cut the seats available for these critical courses, there is now a financial cost to their decision in a way that does not exist currently. Right now, once the enrollment is set, the schools gets the same state revenue regardless of whether they provide courses or not.

A related point was made by Kate Bowles in a Twitter conversation on the bill.

If you have a system that can accommodate students, just not in the courses they need, actually you have a curriculum problem.

This is an excellent, but little discussed, issue in public higher education. Are public institutions offering the right mix of courses and programs based on student needs? As Kate indicates, our problem is not as simple as a course problem – it’s also a curriculum problem.

The challenge, however, is to spark change in our higher education system without having outside parties (such as state government, accrediting agencies, online providers) micromanage what is essentially an academic-led decision on curriculum.

It appears that the backers of SB 520 seek to provide an incentive system that avoids micromanagement – let the academic bodies lead make curriculum decisions – but provides an risk / reward structure to help ensure student needs come first. Should schools decide to essentially outsource part of the lower-division curriculum while providing other courses not in such high demand? Yes, there are reasons to do so in many cases – let the schools decide. But if a schools decides to use its resources this way, reduce the likelihood that admitted students would be short-changed.

Changing dynamics of system online programs

Consider the impact that the recent focus on online education from California government leaders – from Governor Brown’s meetings to the 20mmreboot conference to SB 520 – has already had on the public systems.

The University of California created UC Online nearly four years ago, but the focus prior to 2013 seemed to be mostly on helping the institution find new students and new revenu, and not on helping the already-admitted UC students. As described by the San Francisco Chronicle in 2010:

Long term, the idea is to expand access to the university while saving money. Tuition for online and traditional courses would be the same. But with students able to take courses in their living rooms, the university envisions spending less on their education while increasing the number of tuition-paying students – helpful as state financial support drops.

Last year, CalState announced their online program CSU Online. Like UC Online, the focus was specifically NOT on admitted CalState students, but rather reaching new students and new revenue. In an open letter, the CSU Online executive director called out the goals (and by the way, note that the original letter is no longer available on the CalState site).

The 60 or so fully online self-support programs that currently exist throughout the CSU will comprise our initial effort with an eye toward serving the extensive mid-career professional and unemployed adults who are in need of this level of education to advance their careers. A full listing of the CSU’s online self-support programs is available on the Cal State Online website. The second focus should be the presentation of two or three degree completion programs in an effort to enhance workforce development.

The California Community College system does not have a systemwide online program beyond California Virtual Campus - a portal to find courses and programs offered by individual colleges and districts. However, the various online programs and courses for this system do primarily focus on admitted students.

And today? It is too early to see the effects specifically from SB 520, but we are seeing changes from the general push for online education and the focus on helping students get the courses they need.

From UC Online presentation less than a month ago:

During the UC online presentation, we learned that the university wants to move quickly to place many new courses online starting next Fall. The goal is to rapidly increase the ability of students on one campus to take a course on another campus. There is also the idea that students can take outside MOOC courses and get credit for them by taking an exam or asking for transfer credit. Once again, the stress was on taking care of the gateway course bottleneck.

From CSU Online in an article that just came out today:

Officials plan to use $21.7 million to hire more instructors and student support staff to admit nearly 6,000 more students, $10 million to fund online courses to allow more students to enroll in high-demand, required courses and $7.2 million for incentives for campuses to develop ways to push more students to graduate on time.

While there are still plans to find new students, there is a new urgency to serve admitted students from both programs.

What does success for SB 520 look like?

In this regard, Michael’s point is particularly important.

Let’s start by reminding ourselves of the real goal here. It is not to offer students seats in courses. It’s to get students to complete those courses successfully so that they can graduate more quickly. But there are a number of aspects of the world that SB 520 would create that conspire to reduce the likelihood of achieving that goal significantly. For starters, online courses in general—not just MOOCs—have lower completion rates than traditional face-to-face courses. They require more self-discipline, better reading skills, and better awareness of when to seek help than traditional classes do. Offering an online class to a student who otherwise would be shut out altogether is definitely better than nothing, but we need to recognize that we are already starting with a solution that has its challenges for achieving a goal of high completion rates, even if everything else is equal. [emphasis in original]

There is some real work to be done both during the shaping of the final bill and during the implementation to allow SB 520 to be successful in this context – course completion leading to degrees rather than just course availability. There are some real problems to address, but I’ll leave those barriers as a topic for another post.

I think the very approach of implicitly defining a right for admitted students to have access to the courses they need is significant in and of itself and is certainly worth trying. We need more focus on students.

The post California SB 520 Could Define a New Right Right for Students – Access to Courses appeared first on e-Literate.

Phil has done a great job of covering the news of California’s new bill (or stub of a bill, really) that would create a state-wide system of third-party online courses that would be available to students who would otherwise be shut out of courses that they need to graduate. It’s a good problem to tackle, since it would both make life better for students and improve the long-term state budget situation. Unfortunately, I don’t think the current incarnation of the bill takes into account either the full context and needs of students who find themselves shut out of the core courses or the directions that MOOCs are evolving into. As a result, it offers a bad prescription for the solution. The good news is that the shortcomings can be fixed while remaining well within the spirit of the bill.

What Students Need

Let’s start by reminding ourselves of the real goal here. It is not to offer students seats in courses. It’s to get students to complete those courses successfully so that they can graduate more quickly. But there are a number of aspects of the world that SB 520 would create that conspire to reduce the likelihood of achieving that goal significantly. For starters, online courses in general—not just MOOCs—have lower completion rates than traditional face-to-face courses. They require more self-discipline, better reading skills, and better awareness of when to seek help than traditional classes do. Offering an online class to a student who otherwise would be shut out altogether is definitely better than nothing, but we need to recognize that we are already starting with a solution that has its challenges for achieving a goal of high completion rates, even if everything else is equal.

And everything else would not be equal. Because the courses would not be taught by the faculty of the student’s home institution, there would likely be no opportunity for the teacher to talk the student’s advisor and other instructors, either to gain insights into that student’s needs and problems from people who have already worked with him, or to share information about his needs (e.g., a need for extra tutoring) with his support network. Then there’s the timing. According to the bill, students are only eligible for these third-party courses once it is determined that no such courses are available on their home campus. We don’t know exactly when such a determination would be made, but let’s assume for the moment that it is made at the end of a two-week add/drop period. So, two weeks into the semester, students begin looking outside for new courses. Maybe it takes them another two weeks to find a course, register for it, and begin attending. (We have no idea how easy it will be for students to find and register for these courses.) So now the student is starting an online course four weeks into a fifteen-week semester. That is not a good recipe for success, particularly when the student’s support network is going to be largely out of the loop for the remaining eleven weeks.

And let’s not forget about cost and financial aid. We already saw at the press conference that the bill authors do not yet have a handle on how much these courses would cost. And they didn’t even talk about how the courses would be paid for. Suppose a student has a scholarship at her home institution, but then has to pay $1,000 to a third-party provider for a bottleneck course. Does the home institution have to pay that cost? I’m pretty sure that’s not how scholarship money works now. Changing it to work this way could significantly impact the schools’ ability to give it out. The timing issue I described above could make matters even worse. It’s fairly common for students to go to class without books for the first few weeks of class while they wait for their financial aid checks to come in. If they can’t even file for financial aid until four weeks into the semester because that’s how long it takes them to register for the third-party class that enables them to maintain their full-time status, then they may be held up longer getting their books for every other class. Adding a third-party vendor into the mix inevitably adds bureaucracy. Whenever that happens, there’s a good chance that it will affect the students and their ability to focus on their studies.

So, while SB 520 would probably create a better situation for students trying to get into bottleneck courses than the one they are in now, it may not be a whole lot better in practical terms.

Courses, Courseware, and Course Designs

Interestingly, the one concrete example that Udacity’s Sebastian Thrun gave at the SB 520 unveiling press conference doesn’t fit the model that the bill seems to envision anyway. He talked about SJSU, where local faculty and TAs still teach the course, but they use a Udacity MOOC as courseware. The distinction between a “course” and “courseware” is a blurry one, but basically, if you take the particular instructor out of the course, what you have left is the courseware. If the creator and teacher of a MOOC turns over the keys of the MOOC, with all its videos, assessments, and other materials, to another instructor, then what is being turned over is courseware. If a textbook vendor provides not only the book, but the slides, the lecture notes, and a set of machine-graded tests and homework assignments, organized in a way that a faculty member can adopt without having to modify or supplement it a whole lot, then the publisher is essentially providing courseware. It’s essentially a course in a box that can be used by local faculty to teach local students. And courseware, in turn, is nothing more than a productized course design. If I, as an individual instructor, package up everything I use to teach my course, create videos of my lectures, and write down all the instructions and other details that I usually share informally or keep in my head, then what I have in the package can be called “courseware.” If you think about delivering a course as being like making a meal, then the course design is what’s in the chef’s head and pantry that she combines to make the meal. Courseware is the recipe and box of ingredients provided so that anyone can cook the meal. And a “course” is a particular meal created by a particular chef.

What California needs to overcome the bottleneck course problem in the most effective way possible is not new courses but new course designs and courseware that can be adopted by local faculty to meet the needs of the students they know. It needs recipes for nutritious meals that can be served at scale (like Jamie Oliver’s school food revolution toolkits). It needs new approaches for using technology in the classroom to enable the human instructors to focus on what they do best for more students, but assembled into a polished package that would be straightforward for local faculty to adopt. Such packages could come from a variety of sources. Certainly, MOOC providers and textbook publishers are both good candidates. And, of course, faculty can create these packages themselves, either on their own or with the help of third-party facilitators such as the National Center for Academic Transformation or Lumen Learning.

The variety is important because one size will not fit all. Suppose you have a school that is turning 75 students a year away from a core course. Is the problem that they can’t afford enough teachers or that they don’t have enough classroom space? Are the teachers trained in the technology solution that would help? Does the school have the right equipment? What is the bottleneck subject, and what approaches for scaling that particular subject work. (Math is very different from writing, for example.) Is there a high percentage of at-risk students in the class? Or ESL students? Different answers to these questions will yield different prescriptions for a good solution. The local institution should be both empowered and responsible to solve the problem using their understanding of the details and the full power of the institutional student support network. If some faculty member somewhere has figured out how to teach 1,000 students effectively using a MOOC, and that solution will work effectively for the 75 students at a particular school who are locked out of a course, then chances are good that it will work more effectively for those students if the same course is facilitated and supported by a local faculty member who is on campus, knows their advisors, has been proven to be a “good cook” with experience addressing the local tastes and nutritional needs, and doesn’t have to teach 1,000 students at a shot in order to solve the local bottleneck.

What a Good Bill Would Look Like

In and of itself, the original impulse to require the availability third-party courses is a good and important part of a complete solution. It applies pressure on the schools to come up with better solutions and gives students a better-than-nothing safety valve. Nor would I bend over backwards to accommodate faculty pressure regarding which courses to certify. If students have no chance of completion now because they are shut out of courses, then the primary emphasis of the third-party provisions should be on providing them with chances of completion that are at least incrementally better than zero. The quality standard should be as high as is practical, but the minimum standard of “better than nothing” is…well…better than nothing. The point of the third-party course option is not to have a great solution. It’s to have a better-than-nothing solution when all else has failed. Having this option in place is both right for the students as a last resort and essential as a mechanism to put pressure on the various stakeholders at the schools to solve the bottleneck problem themselves, lest they lose control of the educational experience and, potentially, funding dollars.

But while the third-party course option is an essential backstop, it is far from an optimal solution for the students. The main focus on the bill should be on minimizing the chances that students will be forced to take the third-party courses by mandating, supporting, and funding the development and/or licensing of courseware that empowers faculty to solve the bottleneck problem locally, where that solution can be tuned to the particular needs of the local student population and plugged into the students’ support network at their home institution. Teachers’ unions, for their part, should push to ensure they have the funding and autonomy to take responsibility for solving these problems themselves. They need to be the champions for real and effective change that embraces the possibility of using technology to scale effective education while also being the experts in the room who can distinguish between a real solution and snake oil. They should insist on funding for courseware evaluation, course design development, and faculty training. Rather than fight against the push to use technology to help solve the access problem, they should fight for the ability to lead the change, and to shape it. They should insist on the opportunity to provide better solutions for their students than the third-party option, and then they should prove that they can do it.

The post California SB 520 Currently Misses the Mark, but Not By Much appeared first on e-Literate.

Thanks to Ry Rivard at Inside Higher Ed, we have a copy of the draft bill as of March 8, 2013. With cooperation of Caffeine-Fueled Transcriptions, Inc, I thought it would be useful to share the full text. Please note that this is a draft bill that will be amended over time. I have added formatting (indents) to help with readability. See previous post for description of press conference (including Q&A) and press coverage.

*************************************************************

SECTION 1. The Legislature finds and declares all of the following:

(a) In recent years, California’s public higher education institutions have faced skyrocketing demand for enrollment at a time when they lack capacity to provide students with access to courses necessary for program completion and success.

(b) In the 2012 – 13 academic year, 85 percent of California Community Colleges (CCC) reported having waiting lists for their fall 2012 course sections, with a statewide average of more than 7,000 students on waiting lists per college.

(c) Similarly, impacted courses have contributed significantly to difficulties within the University of California (UC) and California State University (CSU) systems, with figures indicating that only 60 percent and 16 percent of students, respectively, are able to earn a degree within four years, with lack of access to key courses a factor in increased time-to-degree.

(d) With rapidly developing innovation in online course delivery models, California’s public institutions of higher education have a unique opportunity to meet critical demands for enrollment and reduce time-to-degree by providing students with access to high-quality, alternative, online pathways to successfully complete and obtain credit for the most impacted lower division courses.

(e) California could significantly benefit from a statutorily enacted, quality-first, faculty-led framework allowing students in online courses in strategically selected lower division majors and general education fields to be awarded credit at the UC, CSU, and CCC systems. While providing easy access to these courses, these systems could also continually assess the value of the courses and grates of student success in utilizing these alternative online pathways.

SECTION 2. Section 66409.3 is added to the Education Code, to read:

(a) The California Online Student Access Platform is hereby established. The platform shall be administered but the California Open Education Resources Council established pursuant to Section 66409. As used in this section, “platform” means the California Online Student Access Platform established by this section.

(b) The platform shall accomplish all of the following objectives:

(1) Provide an efficient statewide mechanism for online course providers to offer transferable courses for credit.

(2) Create a pool of approved and transferable online courses for credit through which students seeking to enroll may easily access those courses and related content.

(3) Provide a faculty-led process that places the highest priority on educational quality through which online courses can be subjected to high-quality standards and review.

(4) All the state, the public, students, faculty, and other stakeholders to examine student success rates within the platform.

(c) For purposes of accomplishing all of the objectives of the platform as specified in subdivision (b), the California Open Education Resources Council shall do all of the following:

(1)

(A) Develop a list of the 50 most impacted lower division courses at the University of California, the California State University, and the California Community Colleges that are deemed necessary for program completion or fulfilling transfer requirements, or deemed satisfactory for meeting general education requirements.

(B) For purposes of this paragraph, “impacted lower division course” means a course in which, during most academic terms, the number of students seeking to enroll in the core exceeds the number spaces [sic] available in the course.

(2) Create and administer a standardized review and approval process for online courses in which most or all course instruction is delivered online and is open to any interested person. When reviewing online courses for purposes of this section, the council shall, at minimum consider the extent to which each course does any of the following:

(A) Provides students with instructional support and related services to promote retention and success.

(B) Provides students with interaction with instructors and other students.

(C) Contains a proctored student assessment and examination process that ensures academic integrity and satisfactorily measures student learning.

(D) Provides a student with an opportunity to assess the extent to which he or she is suited for online learning prior to enrolling.

(E) Utilizes, as the primary course text or as a wholly acceptable alternative, content, where it exists, from the California Digital Open Source Library established pursuant to Section 66408.

(F) Includes adaptive learning technology systems or comparable technologies that can provide significant improvement in the learning of students.

(G) Includes content that has been reviewed and recommended by the American Council on Education.

(3) Regularly solidity and consider from each of the respective statewide student associates of the University of California, the California State University and the California Community Colleges, advice and guidance on implementation of the platform.

(4) Collect, review, and make public data and other information related to student success within the platform by gathering and reporting data on accepted student success metrics, including, but not necessarily limited to student enrollment in approved online courses through the platform, and student retention and completion rates.

(5) Utilize the state’s current common course numbering system for approved courses so as to simplify the identification and articulation of comparable courses.

(d) Online courses approved by the the California Open Education Resources Council pursuant to this section shall be plead in the California Student Access Course Pool, which is hereby created, through which students may access the courses. Students taking an online course available in the California Student Access Course Pool and achieving ga passing score on the course examination shall be awarded full academic credit for the comparable course at the University of California, the California State University and the California Community Colleges.

The post California SB 520 – Text of Proposed Draft Bill for Online Education Platform appeared first on e-Literate.

Outside of the Vatican, the big news this week in higher education is the proposed legislation in California that would identify and approve a set of up to 50 online courses that the three public systems would accept as credit for admitted students. Senate President Pro Tem Darrell Steinberg is co-authoring SB520 that addresses popular, introductory courses for which students cannot get access from their University of California, California State University, or California Community College campus. The pre-coverage of the event included the following:

• Wall Street Journal, “Push to Widen Online Study in California”
• New York Times, “California Bill Seeks Campus Credit for Online Study”
• Kevin Carey, “California’s Groundbreaking State Online Higher Education Plan”
• Inside Higher Ed, “Outsourcing Public Higher Ed”

As described in the NY Times:

Legislation will be introduced in the California Senate on Wednesday that could reshape higher education by requiring the state’s public colleges and universities to give credit for faculty-approved online courses taken by students unable to register for oversubscribed classes on campus.

If it passes, as seems likely, it would be the first time that state legislators have instructed public universities to grant credit for courses that were not their own — including those taught by a private vendor, not by a college or university.

“We want to be the first state in the nation to make this promise: No college student in California will be denied the right to move through their education because they couldn’t get a seat in the course they needed,” said Darrell Steinberg, the president pro tem of the Senate, who will introduce the bill. “That’s the motivation for this.”

Update 3/14: Text of bill here

Senate President Pro Tem Steinberg held a press conference today via Google Hangout and video streaming (archive available soon on site) to announce the package of legislation. I have attempted to summarize the press conference below (any mistakes in note-taking are my own, and I’ll update as needed). While Michael and I will both have analysis of this proposal soon, I wanted to first share the information directly.

Introductions

Steinberg introduced several key people that were part of crafting the proposal or presenting during the press conference.

• Sen. Marty Block, chair of education sub-committee who introduced companion bill SB547
• Special thanks to 20 million minds & Dean Florez for “help and leadership on these important issues”
• Assemblywoman Christina Garcia, co-author of SB520
• Michelle Pilati faculty senate Community Colleges
• Richard Copenhagen, student rep
• Sebastian Thrun, CEO of Udacity

Why we’re here – Darrell Steinberg

• Steinberg referenced the Master Plan from 1960 as a blueprint for California higher education and a model for the nation. The Master Plan was founded on principle of making higher education available for all regardless of economic means – only to be limited by individual desires / ambitions. The plan led to economic engine of CA and our high-tech position. However, we’re at a cross-roads.
• SB520 as amended “would reshape higher ed” higher ed “in partnership with technology we already use to break bottleneck that prevents students from completing education”.
• California would be the 1st state in nation with such a statewide system.
• No student should be denied education because can’t get course they need.
• What SB520 is NOT:
  • Not substitute for campus-based instruction;
  • Not separated from faculty review – faculty panel can certify up to 50 online courses where students have problems getting into in traditional way. Only for courses where students cannot get into, and only if university is not already offering the course online. The courses can come from anywhere, but have to be approved by 9-member panel of faculty appointed by faculty senates from all three systems; modeled after SB1052/53 on OER access.
  • Not a shift in funding priorities; still plan to invest

Richard Copenhagen

• We face a lack of access with students having to take frivolous units to keep financial aid & transfer requirements; students have gone homeless in cases.
• We won’t solve problems just with online.

Sebastian Thrun

• I am excited about today; shift of higher education in California.
• This bill will empower access to higher education.
• We are technology provider, not educators – leave course design with faculty.

Michelle Pilati

• We are pleased with interest in distance education; excited to see what we can do.
• Technology is adjunct to learning; referenced Richard’s statements – technology is not a solution but can assist.
• We hope we can leverage interest in distance education state of art to better serve students – help them get through courses the first time.
• We plan to work with legislation, shape it into something that will help students.

Marty Block

• Access, affordability & quality – these are the three pillars from Master Plan; over time we have kept affordability (even with tuition raises), and no one questions our quality; however, access is problem.
• What online education does is open up possibilities for students, open up options.
• We need to be careful to maintain quality and academic rigor; with these bills the faculty role is not diminished; courses are reviewed, monitored by faculty.
• I have introduced SB547 – the three faculty senates jointly identify & develop transferable lower-division courses that can be offered online; these will be deemed transferable; works hand-in-hand with SB520 on access.

Christina Garcia

• I was a math professor for 13 years – university & college; saw students struggle to complete courses on time, and even to figure out what’s next.
• I am co-author of this bill (SB520).
• I know first-hand power of online technology.
• One feature is that by providing online courses for those who can benefit from online will free up seats in face-to-face courses for those needing this structure.
• Online education does not replace traditional models – it gives another tool.
• We will create one other bill in this “package”.

Darrell Steinberg

• Here’s what’s happening: Incredible innovation and entrepreneurship on one hand and proud institutions on the other hand; these have been separate worlds.
• If allowed to go separately, we won’t serve students; with the edtech hand, they can’t earn credit for online courses (e.g. MOOCs); on the other hand, with incredible faculty and institutions – we are struggling and there simply are not enough classes to ensure students who want to get their education can do this thru our three systems.
• This bill seeks to put the entrepreneurial innovation and energy alongside the best of higher education system – maintain / enhance quality, ensure process for certifying courses is faculty-driven; also invest more in core education mission.
• This is an exciting day. We would be the first state in nation to figure out how to put the two worlds together. How to certify the best courses, while not only involving faculty but let them take leadership position.

Questions & Answers

• Q. Given budgetary constraints, this could be seen as bringing us into 21st century, yet others may see this as cheapening of education – do you have this concern? A. I would have concern if bill simply said to take outside content and courses and rubber-stamp approval, but that’s not what it does. True partnership between faculty 3 sys and innovators and entrepreneurs. No, I don’t have that concern – we’re providing the correct approach.
• Q. Do these courses cost the same per unit as current system fees? With students signing in remotely, how would you deal with cheating? A. On the fee issue – we still have work to do for specific answer. The general approach – fee should be no more than taking face-to-face equivalent. We also need to make sure savings / revenue is shared in some way by students as well as the universities & colleges. We don’t want to create incentive to reduce face-tof-face instruction and lower cost structure. We will rely on face-to-face proctoring or remote proctoring software for testing req, but we don’t have final answer. Part of faculty panel’s review process is to review feasibility of proposed testing.
• Q. This will clear way for online ed credit in 3 systems. A. Yes, will create smart pathway for certifying online courses for credit.
• Q. Would you have to pass test to get credit at all 3 systems? A. Yes, similar or identical requirement.
• Q. Would students have to be admitted through regular admissions? A. Yes.
• Q. Did you get the same reaction from all three systems? A. I wouldn’t characterize reaction as different. I would characterize as showing interest, excitement, opportunity, and a little or a lot of fear and trepidation. Until we lay the bill out in details, there will need to be some concern. If it wasn’t somewhat controversial, this bill wouldn’t be worth doing.
• Q. How will courses be priced, who will set fees? A. We still have work to do for definite answer. See above.
• Q. Why Google Hangout for press conference? A. This is the first time for this type of event, and it is consistent with policy direction. The world is changing. Technology is important force in our life – mostly a positive force. We want to use technology to help as many young people, students as possible to achieve dreams and compete in modern economy. Using Google Hangout was the right thing to do and is consistent with mission.
• Q. Does the bill outline support for faculty & students? A. We will be very specific for criteria for faculty panel. One of the lead considerations is the extent the course provides opportunity for interaction between faculty and students. If none, the course is not going to be certified.
  • Pilati – It is not completely clear where the bill is going, but student-faculty interaction is critical to success, like current online offerings.
  • Garcia – Keep in mind that with campuses there are multiple support and tutoring systems from institution; weneed to figure out student-faculty interaction, but we do have other methods of support already in place.
  • Thrun – He referenced SJSU pilot with 300 students for-credit, $150 per unity – lower than face-to-face fees; the pilot is staffed with instructors and mentors; have excellent retention rate so far; could shed light.
• Q. Are there 50 courses for each system? A. No 50 courses across all.
• Q. So UC student could take same course as CC? A. Sure, could cross over.
• Q. What is expected cost of this initiative? A. Note that SB1052 /53 appropriated $5M with matching funds – $10m total to launch aggressively. We will look at details as part of  budget process, but we expect the same kinds of numbers, in the ballpark.
• Q. What would be the maximum number of students enrolled in a course? A. First, define population as admitted students. Within this ceiling, we don’t think there would be limit. The issue is whether or not students could get access to face-to-face course. If not, they will be eligible for online course.
• Q. Is this more about saving money or helping students, and any evidence that online education does help students? A. This is about helping students. This is not a substitute for reinvesting in higher education in general. But we will be making a big mistake if we don’t take advantage of technology advances. Students won’t be hindered in ability to get into class. We can get out in front and shape MOOC movement, not just watch it. Without this approach, we actually risk diminishing quality. I admit this bill will be controversial, but we need to get out in front. Use 1052/53 structure to guide us.
  • Thrun – Udacity is only provider pulled classes because of quality concerns, we want quality first.
• Q. What is involvement of Udacity in this proposal? A. Udacity is eligible, like any other provider, to compete before faculty panel. The “good doctor” (Thrun) has been a leader in the field, we need to learn from others. They will be a competitor like anybody else.
• Q. Is Udacity a sponsor, or is there a sponsor? A. There is no sponsor. I gave shout-out to 20 Million Minds Foundation and Dean Florez –  and we worked with them. Steinberg / Garcia are co-authors.
• Q. How many online classes will be allowed to transfer between system and will there be any cap? A. That’s a great question. We don’t have an answer at this point. We need to put this in the mix. Question worthy of consideration.

Additional Coverage

• Mercury News, “Calif. bill would permit online courses for credit”
• Associated Press, “Lawmakers push for online education”
• MSN, “Calif. lawmaker wants to force state colleges to accept online credits”
• VentureBeat, “Online education gets legit: California bill would give college credit”
• Oakland Tribune, “California college students shut out of classes could earn credits online if new legislation passes”
• Chronicle of HE, “California’s Move Toward MOOCs Sends Shock Waves, but Key Questions Remain Unanswered”
• Inside Higher Ed, “Politics and Cautions in California”
• Matthew Yglesias, “California Pointing the Way to Online Education”
• LA Times, “California bill would promote statewide online college courses”
• Campus Technology, “California Bill Could Allow Students To Take MOOCs for Credit”
• Cable Green at Creative Commons, “California Unveils Bill to Provide Openly Licensed, Online College Courses for Credit”
• Times-Herald, “More online courses to ease bottleneck aim of Senate bill”
• Michael Feldstein, “California SB 520 Currently Misses the Mark, but Not By Much”
• LA Times, “Online-course bill is sharply criticized by top UC faculty leaders”
• Inside Higher Ed, “The End Run”
• LA Times, “Give online courses the old college try”
• Inside Higher Ed, “U. of California Faculty Leaders Question Outsourcing Plan”
• Chronicle of HE, “A Massively Bad Idea”
• San Francisco Chronicle, “Faculty spurns online course approval plan”
• Darrell Steinberg in HuffPo, “Breaking the Bottleneck for College Students”

The post Proposed California Legislation for Statewide Online Education Courses – The Basics appeared first on e-Literate.

Analysis of the sketchy available data reveals two myths about MOOCs: First, MOOCs are subsidized, not free. Second, MOOCs have high completion rates for those who seek credit.

To better understand MOOCs, discussions are emerging with classification schemes that provide perspectives of students with different motivations, needs and methods of learning. These schemes can be used to further refine “completion” rates, and to understand subsidies. Though there is limited data available, the several examples can be used to develop a better, but far from complete, understanding of MOOCs in higher education.

Colleges and universities need to know the full costs of offering “free” courses, even when provided by a commercial firm to students at no cost. The definition of “completion” becomes critical to public funding policies for students and colleges and universities

The post MOOCs: Student Motivation, Perspectives and Government Incentives appeared first on e-Literate.

Well, it’s that time of year again. On April 15th, GSV Advisors‘ annual Education Innovation Conference will kick off. Reactions to the conference last year were a bit of a Rorschach Test, varying greatly depending on the person’s view of the role of venture capital in education in general (as well as whether they were attending the conference physically, watching remotely, or just reading the tweet stream). As I wrote about after last year’s conference in “What Are Ed Tech Entrepreneurs Good For?“, I am in the “cautious but positive” camp about the importance that venture funding could play in education.

Love it or hate it, this year’s conference is going to be bigger than ever. There are going to be 180 companies presenting, which is double the number from the previous year, and the conference organizers predict a 50% increase in attendance when all last-minute and walk-in registrations have been tallied. I will be attending again this year and expect to be able to give more thorough coverage than I was last year, thanks in part to the new freedom that comes from working at MindWires.

Anyway, one of the more interesting features of the conference this year is the Return on Education (ROE) awards. It’s basically a contest to recognize up-and-coming ed tech projects. There is no monetary award, but given that the conference is being heavily attended by education, business, and general news press, the winners can expect a lot of good PR and, likely, a lot of attention from people with money. There is a non-profit category for the awards, so it’s not just limited to start-ups. Projects have to be nominated by attendees, and there are 100 such nominations so far.

Nominees are judged on the following criteria:

• Significantly increase access to education;
• Greatly reduce the cost for learners and/or learning institutions;
• Dramatically improve learning outcomes;
• Provide substantial leverage to learning leaders (teachers, professors); and/or
• Make a sustainable and scaled impact.

The nomination form has fields asking for explanations on how the nominee fits each of these criteria.

Nominations are open until Friday. I’ll be curious to see what kinds of projects get nominated as well as what kinds of projects win.

The post The 2013 Ed Innovation Return on Education Awards appeared first on e-Literate.

In part 1 of this series of posts on MOOC student patterns, I shared an initial description of four student patterns emerging from Coursera-style MOOCs based on new data from professors. In part 2, I revised the description based on some feedback and added a graphical view. The excellent feedback has continued, primarily through comments to both posts mentioned above as well as a separate Google+ discussion. This process has helped identify a fifth pattern, clarify the pattern description, and improve the associated graphic. In particular, I want to thank Debbie Morrison, Colin Milligan, John Whitmer, Charles Severance and Kevin Kelly – as well as other commenters for the great discussion.

The primary changes involve clarifying the previously-described Lurker category. I have separated out a new No-Show category and renamed the Lurkers as Observers. There are also tighter descriptions of each pattern that help define potential data collection that would identify these groupings. Here are the new descriptions and updated graphic.

No-Shows – These students appear to be the largest group of those registering for an Coursera-style MOOC, where people register but never login to the course while it is active.

Observers – These students login and may read content or browse discussions, but do not take any form of assessment beyond pop-up quizzes embedded in videos.

Drop-Ins – These are students who perform some activity (watch videos, browse or participate in discussion forum) for a select topic within the course, but do not attempt to complete the entire course. Some of these students are focused participants who use MOOCs informally to find content that help them meet course goals elsewhere.

Passive Participants – These are students who view a course as content to consume. They may watch videos, take quizzes, read discuss forums, but generally do not engage with the assignments.

Active Participants – These are the students who fully intend to participate in the MOOC and take part in discussion forums, the majority of assignments and all quizzes & assessments.

 

The post Emerging Student Patterns in MOOCs: A (Revised) Graphical View appeared first on e-Literate.

I almost never quote a blog post in its entirety, but this one from Dan Meyer is so good that I just can’t bear to cut a single word:

Stephanie Simon, reporting for Reuters on inBloom and SXSWedu:

Does Johnny have trouble converting decimals to fractions? The database will have recorded that – and may have recorded as well that he finds textbooks boring, adores animation and plays baseball after school. Personalized learning software can use that data to serve up a tailor-made math lesson, perhaps an animated game that uses baseball statistics to teach decimals.

Three observations:

One, it shouldn’t cost $100 million to figure out that Johnny thinks textbooks are boring.

Two, nowhere in this scenario do we find out why Johnny struggles to convert decimals to fractions. A qualified teacher could resolve that issue in a few minutes with a conversation, a few exercises, and a follow-up assessment. The computer, meanwhile, has a red x where the row labeled “Johnny” intersects the column labeled “Converting Decimals to Fractions.” It struggles to capture conceptual nuance.

Three, “adores” protests a little too much. “Adores” represents the hopes and dreams of the educational technology industry. The purveyors of math educational technology understand that Johnny hates their lecture videos, selected response questions, and behaviorist video games. They hope they can sprinkle some metadata across those experiences — ie. Johnny likes baseball; Johnny adores animation — and transform them.

But our efforts at personalization in math education have led all of our students to the same buffet line. Every station features the same horrible gruel but at its final station you can select your preferred seasoning for that gruel. Paprika, cumin, whatever, it’s yours. It may be the same gruel for Johnny afterwards, but Johnnyadores paprika.

Dan captures most of what I was trying to get at with my rant on the big data hype, but much more clearly and succinctly. Points two and three are the most salient here. First of all, the sort of surface-level analysis we can get from applying machine learning techniques to the current data we have from digital education system is insufficient to do some of the most important diagnostic work that real human teachers do. Think about the math classes is which you had to show your work on your homework. Why was that important? Because the teacher needs to see not only what you got wrong but why you got it wrong. Teachers generally don’t just say, “You got three out of five problems involving converting decimals to fractions wrong. Go study some more.” They sit down and work through the problems with the student to find the source of the errors. It’s really hard to get computers to do this well, even with highly procedural domains like math. (Forget about, say, literary analysis.) So in the vast majority of cases, we don’t even try to design systems where students show their work. And without the step-by-step data, no fancy algorithm is going to teach Johnny.

Second, if the problem is that your content isn’t what the student needs, no fancy algorithm is going to fix that either. Videos are a prime example. I know of one textbook publisher whose teacher customers report that students won’t watch the publishers’ videos, but they can and do find videos on the same topic on YouTube and share them with each other. Think about that. Video-based pedagogical support is valuable enough to the students that they will expend energy searching for videos and sharing them. But they reject the expensive, carefully crafted videos from the publisher that are served up to them on a silver platter. It’s not that the publisher-supplied videos are necessarily “bad” in the sense that they have poor production qualities or are unclear or factually inaccurate. But the students have a particular use in mind for the videos. Maybe they’re struggling with a particular homework problem and just need a quick walk-through of a technique so that they can see the step that they are missing, for example. If the video doesn’t fit their needs—both utilitarian and aesthetic—then it won’t get used. Serving it up adaptively isn’t going to help that problem.

That said, it’s worth taking a little time to break down the different types of adaptive learning analytics into a couple of categories and see just what we should and should not reasonably hope to gain from them.

Topological Analytics

The first category of adaptive analytics are what I call “topological,” by which I mean that they look at the surface characteristics of whatever data is in the system. They don’t assume any special content tagging, or really any understanding of the content at all, and they don’t require any special tweaks to the user interface of the software. They’re just sifting through whatever data is already in the system and mining it for insight. Within this genus, there are several species of adaptive analytics.

Early Warning Systems

Most people don’t think about early warning systems as being in the same category as adaptive analytics, but if you consider that “adaptive” really just means “adjusting to your personal needs,” then a system like Purdue’s Course Signals is, in fact, adaptive. It sees when a student is in danger of failing or dropping out and sends increasingly urgent and specific suggestions to that student. It does that without “knowing” anything about the content that the student is learning. Rather, it’s looking at things like recency of course login (Are you showing up for class?), discussion board posts (Are you participating in class?), on-time assignment delivery (Are you turning in your work?), and grade book scores (Is your work passing?), as well as longitudinal information that might indicate whether a student is at-risk coming into the class. What Purdue has found is that such a system can teach students metacognitive awareness of their progress and productive help-seeking behavior. It won’t help them learn the content better, but it will help them develop better learning skills.

Black Box Analytics

Then there are those systems where you just run machine learning algorithms against a large data set and see what pops up. This is where we see a lot of hocus pocus and promises of fabulous gains without a lot of concrete evidence. (I’m looking at you, Knewton.) But if you think about where these sorts of systems are employed successfully, it will give you a sense of how they can help and what their limits are likely to be. Probably the best example that I can think of are the recommendation engines that we see from companies like Amazon or Netflix. It’s easy to imagine a system that can tell you, “students who watched this video did better on the test”, or even “students who found this video helpful also found these other videos helpful.” That could be valuable. But there are a few caveats. First, there is vastly more noise in educational data than there is in shopping data. Particularly in face-to-face classes, the system has absolutely no insight into what happens in class. And even in fully online classes, unless they are massive and therefore taught essentially the same way to many students, you’re just going to get a huge amount of variation across many variables that may or may not be relevant and may or may not be independent of each other. It’s I’d tough to find the signal in all that noise. Second, this sort of approach will only work if there is, in fact, a video (or other content element) that makes a difference in learning outcomes in the set of content that students are seeing. If the content is not sufficiently effective to have that kind of an impact (for whatever reason), then machine learning will tell you nothing. And finally, even if you do find content (or behaviors) that are particularly impactful on learning outcomes, the machine can’t tell you why they are impactful. In the best case, they can highlight the effective resources or learning experiences when it finds them, but they can’t ensure that we are crafting impactful resources and learning experiences in the first place. At best, they can point humans to correlations they should look at for clues.

Zombie Learning Styles Analytics

One example that we hear a lot—and that Dan mentions in his post—is the one of the kid who learns better from videos (or “adores” video). This is hardly new. Research into learning styles has been going on for decades. And guess what? Nobody’s been able to prove that any particular theory of learning styles is true. I think black box advocates latch onto video as an example because it’s easy to see which resources are videos. Since doing good learning analytics is hard, we often do easy learning analytics and pretend that they are good instead. But pretending doesn’t make it so. Is it possible that machine learning will turn up statistically significant evidence that some students learn better with videos, (or short videos, or audio with female narrators, or whatever)? Yes, it is. But I will believe it when I see it.

Semantic Analytics

Separate from the topological adaptive analytics are the semantic analytics, where the adaptivity depends on having some understanding of the content. Unsurprisingly, these types of analytics are harder than topological analytics, because they require the content to be tagged in a way that is useful to the analytic engine. But they can be very effective.

Adaptivity by Learning Objective

This is the one that all the publishers are focused on right now. Basically, you form a bunch of what I call golden triangles—learning objectives linked to related learning content and learning assessment activities. The system flags which learning objectives you need to work on based on your assessment and points you to the related content. Obviously, this can be a time-saver by helping students to skip work that they don’t need to bother with. It also can help students who don’t have good meta-cognitive skills become aware of what they don’t know. And it certainly can help teachers understand what skills their class is struggling with, which students might work well together because they’re on the same level (or because they’re not), and so on. But again, as Dan pointed out in his post, this level of adaptivity won’t actually teach students the concepts they are struggling with. All it will do is focus them on the topics that they need to learn.

“Inner Loop” Adaptivity

I have written about the inner loop before. (I strongly recommend reading Kurt VanLehn’s paper on Intelligent Tutoring Systems for an overview of this and related topics.) Basically, this is the “show your work” monitoring that I talked about at the top of the post. Intelligent Tutoring Systems do what the adaptive-by-learning-objective systems do, but at a micro level within a problem that the student is trying to solve. (Hence, “outer loop” and “inner loop.”) They flag the step that the student gets wrong and offer hints related to that particular step to help the student get back on track. Carnegie Mellon’s OLI and Carnegie Learning are probably the two most widely known examples of products with inner loop adaptivity. There is decent and growing empirical evidence that these systems work to teach students, but they are time-consuming and therefore expensive to design, require relatively high levels of skill, and we don’t really know how to do them well for knowledge domains that aren’t either highly procedural or highly fact-driven (like writing, for example).

Adaptivity by Activity

This is one of the most interesting and least well-developed areas of adaptive analytics. Rather than adapting content, the idea is to adapt activities. There are examples of this for lower-level cognitive domains like memorization and related language-learning skills. What we don’t have yet is a good vocabulary of the kinds of classroom moves that teachers make in order to match them up against different learning contexts and learning outcomes. This is a topic that merits its own post.

The post A Taxonomy of Adaptive Analytics Strategies appeared first on e-Literate.

Update (3/10): Patterns and descriptions have been updated based on feedback in a new post. Added links to Astronomy, AI Planning courses.

Thanks to feedback from my last post, I have modified the proposed description of patterns for students engaged in MOOCs. I also want to introduce a graphic to visually represent these patterns.

• I have removed the language comparing passive participants to traditional students based on the idea that they expect others to define academic goals for them and ‘expect to be taught’ (thanks to Colin Milligan for description). While this distinction between passive and active participants is important, I have removed the direct reference to traditional students – the reader can apply their own comparisons.
• I have removed the usage of the term archetype. As Satia Renee put it so well on Google+, archetype implies “more an internal personality type expressing itself in patterns of behavior” when I am trying to capture the patterns of behavior. Thus I’m sticking with the less loaded term of patterns.
• I have added language, thanks to Kevin Kelly, that captures the growing case of Drop-Ins as students focused on a particular topic within a MOOC for usage outside of that MOOC.
• Finally, I have moved Drop-Ins right after Lurkers based on Colin’s comments and to help with the graphical view below.

As a recap, I believe we are seeing the following four patterns of student behavior within MOOCs:

• Lurkers – These students are the majority of xMOOC participants, where people enroll but just observe or sample a few items at the most. Many of these students do not even get beyond registering for the MOOC or maybe watching part of a video.
• Drop-Ins – These are students who become partially or fully active participants for a select topic within the course, but do not attempt to complete the entire course. Some of these students are focused participants who use MOOCs informally to find content that help them meet course goals elsewhere.
• Passive Participants – These are students who view a course as content to consume and expect to be taught. These students typically watch videos, perhaps take quizzes, but tend to not participate in activities or class discussions.
• Active Participants – These are the students who fully intend to participate in the MOOC, including consuming content, taking quizzes and exams, taking part in activities such as writing assignments and peer grading, and actively participate in discussions via discussion forums, blogs, twitter, Google+, or other forms of social media.

An important point is that some students change between patterns – such as a passive participant deciding to fully jump in and become an active participant, or even an active participant becoming frustrated and becoming a lurker. From what I’ve seen, this type of change occurs once per course at the most for any individual student.

There are still some good comments coming into the original post, so I will probably refine the definitions over time.

Additional Notes on Behavior

I have already described the data captured by Katy Jordan on MOOC completion rates. Note that this data compares the ratio of students completing a course to total number of students registered.

There are several courses *, typically on Coursera, where we now have a deeper description of the student behavior based on information shared by the professors. I suspect this view will be different between xMOOCs and cMOOCs, and even between different MOOC providers. For now, treat these observations as primarily based on Coursera-style MOOCs.

The majority of students (60 – 80%) reported as registered in a course are lurkers who tend to leave the course completely by the second week and may not even engage with the material in any significant way.

• IHTS: 46k registered, by week 2 there were 11.6k who “completed week 1″
• EDC MOOC: 46k registered, by week 2 there were 7.4k logging in
• Bioelectricity: 12k registered, 8k watched any videos, 3k watched week 2 intro
• Microeconomics: 37k registered, 25% watching video during week 2

The courses seem to stabilize by week 2 or 3 in numbers of students still in course. While this observation is mostly based on anecdotes from blog posts, there are two charts capturing the data for Bioelectricity and EDC:

 

 

Graphical View

How would these student patterns appear over time, at least for those courses similar to the Coursera MOOCs with intermediate data? I believe the following graphic captures the basic shape and topology of student patterns. Note the graphic is a generalization, and ideally we would have this type of diagram for each course.

 

Link to Google+ post

* Links to information on specific MOOCs with intermediate data provided by professors:

• Internet History, Technology and Security
• Screen Capture of EDC MOOC data
• Report on Bioelectricity MOOC
• Post on Katy Jordan Completion Data
• Microeconomics Discussion (see Jan 30 announcement)
• Intro to Astronomy
• Artificial Intelligence Planning

The post Emerging Student Patterns in MOOCs: A Graphical View appeared first on e-Literate.

In my last post, I described a vision for combining elements of MOOC-like scale with a more traditional face-to-face classroom experience, as articulated by Pearson’s Adrian Sannier. (Full disclosure: Pearson is a client of MindWires Consulting.) A couple of months ago, I suggested in an interview with Josh Kim for Inside Higher Education that this is where MOOCs would go next:

Question 6. Will MOOCs replace accredited curriculum? Why or why not?

I don’t know how to answer this question, because I’m not convinced that we know what a MOOC is yet. Will there be massive elements that are integral to many curricula? Almost certainly, although I don’t know how much of it we will see in 2013. Will the curricula be all massive? Probably not in most cases. Will we consider the mix of massive and non-massive elements to be “MOOCs”? I don’t know.

The problem that we have right now is that we have very few models for how this might work. So when Sannier mentioned a course called Habitable Worlds being developed by Professor Ariel Anbar and Lev Horodyskyj at Arizona State University that will eventually be brought to OpenClass to support this model, I asked to speak with Professor Anbar in order to get some specifics. The conversation shed some light not only on possibilities for the mixed model, but also on possible futures for the liberal arts and the role of the professor.

A Quest for Intelligent Life in the University

The core organizing principle of Habitable Worlds is something called the “Drake Equation,” formulated by astrophysicist Frank Drake during the early days of Search for ExtraTerrestrial Intelligence (SETI) as a discipline. Anbar describes it as “a way of organizing our ignorance about what we need to know in order to figure out whether there is intelligent life in the universe.” It has seven variables, ranging from things we know quite a bit about, like the rate of star formation in our galaxy per year, to things we know something about, like the average number of planets that can potentially support life per star that has planets, to things we know almost nothing about, like the average lifespan of a technically advanced civilization. Through a series of interactive exercises, the students learn about each of these variables. They are also each given a unique, computer-generated star field. Their ultimate quest for the course is to come up with an estimate of the number of planets within that star field with which communication might be possible (which is the result of the Drake equation).

Click here to view the embedded video.

It’s fair to say that Habitable Worlds is an attempt to rethink Gen Ed. It’s designed to satisfy the quantitative general science requirement, counts for laboratory credit at ASU as a fully online course for non-science majors. Phil Regier, Dean of ASU Online, gave Anbar permission to start from a blank sheet of paper (and, apparently, something relatively close to a blank check). For his part, Professor Anbar confessed his interest was in “teaching science as a process, not a bunch of facts.” “I wanted to design a course around a quest or a game-like concept where you have to solve a problem,” he continued. “A good lab does exactly that.”

Let’s pause here for a moment and consider the implications of this statement. The rap against liberal arts in the current political environment is that students don’t learn skills and knowledge that are useful for employment. For the moment, let’s leave aside the question of whether that is the only or the most important purpose of higher education. Let’s focus instead on the question that the critics raise. Is it true that liberal arts fail to teach job skills? And to the degree that it is true, is it inherent in the notion of the liberal arts education, or is it due to the way we currently structure the college experience? “Liberal arts” tends to be interpreted as shorthand for taking music and sociology and art history and a bunch of other topics offered in the smorgasbord that is today’s undergraduate course catalog. It is often taken to mean breadth of knowledge producing a “well-rounded” person. But Professor Anbar offers a very different notion of liberal arts here. Habitable Worlds is a science class for non-science majors designed to teach them how to solve problems like scientists do. Is this skill set one that would be useful in the workplace for people in non-science careers? You bet it would. The work of liberal arts is done not by running through a checklist of knowledge that a student should be exposed to, by crafting an experience that is focused on enabling the students to acquire skills that a student may learn from exposure to a particular discipline. That change in emphasis requires a different design both of the curriculum or “major” as a whole and of individual courses. A good lab teaches problem solving. But how many labs are designed and taught with that goal in mind? What are the odds that a typical liberal arts student in a typical liberal arts program is going to be deliberately and thoughtfully exposed to the problem-solving skills that science can teach? If the liberal arts are to survive, the answer needs to be “close to 100%”.

Anbar also talks about creating a self-sustaining knowledge community among the students. That is a core course goal, and implicitly, a core skill set that students need to learn. In this regard, he thinks that the standard survey-level courses are particularly bad candidates, because they don’t naturally stimulate the kind of interest and discussion that what he called “integrative curricula” like climate science or SETI does. So again, this suggests a need for a shift in emphasis in liberal arts from content to skills. Some of this has already been happening for quite a while, piecemeal, through Gen Ed and core competency efforts, but there hasn’t been a real wholesale shift in mindset yet.

Course Design

For the expository portions of the course, there’s heavy use of video and interactive media. For example, there’s a “virtual field trip” to Shark Bay, Australia, where students “walk around” the beach and explore various icons, some of which include David Attenborough-like narrations from Anbar. The exercises are built on a platform called SmartSparrow. (More on that in a bit.) One of the more interesting aspects of the course design is the twist on the use of robo-grading that comes when you think about it in terms of a game. The students are trying to make progress in a challenge game, which essentially robo-grades their exercises. “Assessment is done by the simulators, like levels in a game,” says Anbar. “There are a huge number of A’s, even though there was a lot of anxiety, because if you put in enough time, you can make it through. Teacher and TAs mainly decide what the grade boundaries are and fix things when they don’t work.”

Anbar acknowledges that building a course like Habitable Worlds is “expensive.” (The scare quotes are his.) This is where scale comes in. If he were only developing the course design for his own students, the cost couldn’t be justified. But if it becomes a Gen Ed course that all ASU students go through, or if it is adopted by schools other than ASU through Pearson’s OpenClass or some other platform, then the cost can be amortized. This is a different emphasis on the benefits of scale than you get from MOOC providers, where they talk about the benefits of scale in not only production but also delivery. Anbar’s view of scale in delivery, which I’ll get to in more detail later in this post, is more nuanced. But he definitely stresses the value of scale through re-use as a way to justify production expenses.

At the same time, he pointed out some challenges with current tool sets. He sees the biggest cost challenge as the authoring platforms that enable the development of the interactive components. Today, he claims, developing good game- and simulation-based courses is difficult and expensive because “everything you do has to go through the hands of a sophisticated programmer. We all know that as teachers, we’re always iterating and improving, at least if we’re any good.” So working with developers to hand craft simulations is not a good fit, in his view. That’s why he was excited to find the SmartSparrow authoring tool, which he compared to PowerPoint in terms of how it might be used by instructors in the future. “”Five years from now,” he said, “the question will be ‘What’s your authoring platform,’ not ‘What’s your LMS?’ The LMS is your file cabinet or your book shelf. It’s the most boring part of the equation.”

Overall, he was fairly disparaging of traditional LMSs, specifically calling out their “disappointing” and “limiting” discussion engines. (Habitable Worlds uses Piazza.) He liked the idea of OpenClass because “Pearson is designing OpenClass so that it can basically get out of the way. I can have my course experience to take over the entire screen. You don’t want students seeing everything through the lens of the file system.” That said, there is no date yet for when Habitable Worlds will be available on OpenClass because some of OpenClass’ APIs are not ready yet. “Where waiting for OpenClass to be more Open,” said Anbar.

The Role of the Instructor and Scale in Delivery

So where does the instructor fit into all of this? One of the criticisms of the xMOOC 2.0 model, where instructors act as on-site facilitators to support the massive course experience, is that it “reduces instructors to TAs.” I’ve never fully understood what that criticism means, but I wanted to know what Anbar thinks the role of the instructor is—and isn’t—in his model. His answers were interesting. First, he is quite happy to let go of as much of the grading function as he ethically can. ”I’ve turned myself into an expert guide and a coach,” he said. “That’s what I want to be as a teacher, so that’s what I’ve done. I’m not interested in being a judge.” He spends a lot of his course time on the discussion boards. We talked about the difference between the instructor and the TA. So far, he’s taught the course with up to 500 students in it and has a goal of 1000, so TAs are definitely involved. The difference in role he described is subtle and has everything to do with the perception of him by the students as an expert. The TAs are there to coach, but the students respond differently to somebody that they perceive to have deep experience who is there to support them in their course quests. It’s not too far from what old school asynchronous learning researchers might call “teaching presence.” Some of that comes from interactions on the discussion board, but some of it is about the way in which he is presented in the course materials. He was very aware of his voice (literally) as the voice of the course. As he expands the course out to be taught by other professors, he is thinking about either genericizing the videos or making it possible for instructors to put in their own. He also spoke about maybe making the course a little more modular so that professors have more control over the shape of the curriculum, although he didn’t offer any specifics for how this could be done in a course with such a strong organizing principle.

I must say that Anbar was substantially more agnostic about scale than Sannier was. He definitely was imagining the possibility of the course as a MOOC-like “self-organizing online community.” That said, his focus, at the moment, is on the course as a large, but not MOOC-large, online course in which a human instructor still plays a very substantial role as a facilitator. But the role of the instructor is very substantially different from in traditional courses, whether face-to-face or online. Between that and the focus on teaching skills in an interdisciplinary, quest-driven context, Habitable Worlds is a pretty radical experiment.

Update: Professor Anbar has asked me to point out that his co-creator of the Habitable Worlds class was ASU staff member Lev Horodyskyj. He wrote, “It’s fair to say that while the vision is mine, [Mr. Horodvskyj] has had the larger hand by far in the implementation, operating as a sort of super instructional designer and, more recently, course manager. And, as in any creative partnership, there’s been a sufficient melding of minds that I consider him a co-creator of the course.” Accordingly, I have added Mr. Horodvskyj’s name to the introduction of the post. Anbar also asked that credit be given to NASA’s Astrobiology Institute, “ which helped catalyze some of this (especially the virtual field trips).”

 

The post The OpenClass Vision: An Example appeared first on e-Literate.

Update (3/10): Patterns and descriptions have been updated based on feedback in a new post.

As discussed in my last post, the focus on “completion rates” in MOOCs is somewhat misplaced, as open education is not simply an extension of traditional education. As several others have noted, not every student is attempting to complete a course, and in fact different students have different goals while participating in the same open course. This holds true for both cMOOCs and xMOOCs.

Does this mean that we should throw out the completion rate data? No. As Katy Jordan described quite well in the comments:

A lot of people have asked whether completion rates are the right way of framing the success of a MOOC; I agree that there is much more to the potential positive impacts of MOOCs for students than completion rate but, at the moment, completion rate is what the providers are measuring most consistently.

In my mind, we should augment the models we use to evaluate MOOCs rather than throw the baby out with the bathwater. The challenge, therefore, is to move beyond the simplistic view of one type of student with one type of goal (course completion), and find patterns of student behavior that will give additional insight into the different goals and therefore different measures we should have in evaluating whether MOOCs are effective.

Study Based on Change11

In 2011-2012 as part of the Change11 course (a connectivist course, or cMOOC, facilitated by George Siemens, Dave Cormier and Stephen Downes), the Scottish group Caledonian Academy was given access for surveys and follow-up interviews to help understand the student population in a research study.

The first component of the study was to ask participants to complete an SRL profile instrument* we had developed for the study. The instrument was adapted from a number of pre-existing SRL self-report  instruments (full details, and a copy of the instrument are here), most notably the Motivated Strategies for Learning Questionnaire (Pintrich et al 1991) and a more recent Self directed Learning Orientation scale developed by Raemdonck (Gijbels et al , 2010). [snip]

We saw different patterns of engagement. In addition to an expected cluster of lurkers who purposefully did not engage with other course participants, we identified two further groups: one group of passive participants, who expected ‘to be taught’, and viewed the course as a source of information, attempting to capture all the ideas being exchanged within the Change 11 community; and a final group, more active participants, who set their own goals, established connections with other learners and linked these connections with their existing personal learning network. [emphasis added]

Based on various first-hand descriptions of MOOCs over the past year, I would propose a fourth pattern – the Drop-In, where students who direct most of their active participation for a particular topic within the course or a particular discussion thread.

The Four Student Archetypes

This leaves us with four student archetypes to consider (note that these are emerging patterns based on partial information, and these descriptions may need to change as we get more data):

• Lurkers – This is the majority of students within xMOOCs, where people enroll but just observe or sample a few items at the most. Many of these students do not even get beyond registering for the MOOC or maybe watching part of a video.
• Passive Participants – These are students who most closely align with traditional education students, viewing a course as content to consume. These students typically watch videos, perhaps take quizzes, but tend to not participate in activities or class discussions.
• Active Participants – These are the students who fully intend to participate in the MOOC, including consuming content, taking quizzes and exams, taking part in activities such as writing assignments and peer grading, and actively participate in discussions via discussion forums, blogs, twitter, Google+, or other forms of social media.
• Drop-Ins – These are students who become partially or fully active participants for a select topic within the course, but do not attempt to complete the entire course.

These are not static patterns, in that students may move from one archetype to another. Lurkers may decide that they should spend more time in the course and become passive participants. Passive participants may become more engaged and become active participants over time. Of course, any of these students may also drop out and leave the course.

These student archetypes generally have different goals. Lurkers may not have specific goals beyond finding out what the course is about or doing a “drive-by” evaluation of whether the course merits more time and attention. Passive participants, as discovered in the Change11 MOOC may desire to just experience the MOOC platform or course design.

One problem with our study which we hadn’t anticipated (but perhaps should have) was that individual participants might have quite different (conflicting?) reasons for signing up. While some participants signed up for the content of the course, others (the majority) were primarily or exclusively interested in experiencing the Change 11 MOOC as a learning environment, often because they wanted to implement some of the features of a MOOC in their own practice.

I should also note that while the student archetypes are somewhat based on the general goals for taking a course, there are also important, but largely unexplored, questions on why students leave a MOOC. As Laura Gibbs described in several Google+ discussions, leaving a course because you got what you wanted is very different than leaving due to abusive discussion forums.

Whither Completion Rates

How would our understanding change if we understood the different student archetypes and goals for enrolling in MOOCs? I believe we would end up with better feedback to improve the MOOC models, and a more realistic discussion about the impact of MOOCs. Katy’s data curation and visualization is based on the data available, which is invaluable, but I think her linkage to sources might give us insight to build on the prevailing model and more closely understand student goal completion.

Completion rate should really be measured for active participants. For those students who planned to complete the course and participate in all or most activities, how many ended up achieving that goals and completing the course?

Let’s consider Internet History, Technology and Security taught by Charles Severance in 2012. By traditional measures (as captured by Katy) there were roughly 46k students enrolled with 4.6k students who received a certificate, leading to a completion rate of 10%.

But look  a little closer at the data using Katy’s links:

There were 11.6k students who completed the first week of activities – a rough measure of active participants. Using the four student archetypes, the completion rate was closer to 40%. Likely the rate was higher as the 11.6k number included Drop-Ins who did not intend to fully participate in the full course. But for now, we don’t have the data to accurately separate out this group.

To me, these measures of 11.6k students who actively participated with 40% completing the course is more meaningful than the 46k students enrolled and 10% completion rate. Clearly the majority of the 46k never intended to participate in the whole course. In a traditional face-to-face course, would we include all students who checked out a course syllabus or students auditing a course as actual students in the completion rate measurements? No, we would only count students who indicate through the add / drop period that they intend to fully take the course.

I’d appreciate feedback on these patterns – feel free to comment below or in the Google+ post.

The post The Four Student Archetypes Emerging in MOOCs appeared first on e-Literate.

How many times have you heard the statement that ‘MOOCs have a completion rate of 10%’ or ‘MOOCs have a completion rate of less than 10%’? The meme seems to have developed a life of its own, but try to research the original claim and you might find a bunch of circular references or anecdotes of one or two courses. Will the 10% meme hold up once we get more data?

While researching this question for an upcoming post, I found an excellent resource put together by Katy Jordan, a graduate student at The Open University of the UK. In a blog post from Feb 13, 2013, Katy described a new effort of hers to synthesize MOOC completion rate data – from xMOOCs in particular and mostly from Coursera.

Via a combination of thinking about ‘what makes a successful MOOC?’, and looking for a topic for my final project on the Infographics MOOC, I decided to try to pull together the various statistics floating around online about MOOC completion rates. I’m trying to see if any differences emerge on the basis of platform or the assessment methods used.

My draft graph synthesising everything I’ve found so far can be found here: http://www.katyjordan.com/MOOCproject.html

What Katy has done so far is compile completion rate data from 24 MOOCs – 19 from Coursera, 3 from edX, 1 from Udacity and 1 from MITx (a precursor to edX). The data is available in a table format or in a graph.

 

As you can see on the right side, you can filter by MOOC provider, University or Assessment Type (auto grading, peer grading or some combination).

In the graph view, you can see completion rates vs. total enrollment of each MOOC (the colors represent different Assessment Types).

 

By clicking on each dot, it highlights the MOOC, basic data and link to data source.

 

This is the most thorough summary of MOOC completion rate data that I have seen – for xMOOCs. Kudos to Katy for putting together this data, and please comment on her blog post to let her know of new data available.

Some notes based on the data available:

• The average completion rate of xMOOCs is 7.6%, with a minimum of 0.67% and a maximum of 19.2%. The 19.2% appears to be an outlier from Ecole Polytechnique Fédérale de Lausanne, although it may be worth figuring out how they got their rate so high.
• Does it bother anyone that we get this data from a graduate student from The Open University but not from any of the xMOOC providers who claim the power of data analytics in their platforms?
• In the end, the meme of 10% MOOC completion rates is not too far off, at least for Coursera courses.

The problem I see is that scalar metrics of # of students and completion rates are insufficient for open courses, where there are different student types. Only a subset of students actually plan to complete a course, whereas the majority of students enrolled are there only to sample and explore. Measuring completion rate off the total number of students is misleading, but so is touting the total number of students enrolled (billions served) as a key metric. This is not a problem with Katy’s data, but with the common discussion of MOOCs in popular media and even higher education circles. I’ll address a different model in my next post.

Discuss here or at Google+ post.

The post The Most Thorough Summary (to date) of MOOC Completion Rates appeared first on e-Literate.

When Pearson’s OpenClass was announced about a year and a half ago, the natural question to ask was whether it would disrupt the LMS market. But that was then and this is now. (Full disclosure: Pearson is a client of MindWires Consulting.) The more interesting question today is where OpenClass stands vis-à-vis the MOOCs. To begin with, it certainly appears that the LMS and MOOC markets may be on a collision course. But beyond that, Pearson is a content provider first and foremost. With both the content and the platform at their disposal, as well as an array of assessment tools, they certainly have all the raw materials to build MOOCs. Is that where OpenClass is going? I recently had a short conversation with Pearson’s SVP of Product, Adrian Sannier, to find out. And the answer appears to be, “Not exactly.”

Even within the xMOOC crowd, there has been a stepping back from talk of disrupting college and more effort to show how MOOCs and traditional colleges can co-exist peacefully. Witness, for example, Sebastian Thrun’s talk about “MOOC 2.0″ including teacher facilitation and the pilot of just such a program between Udacity and SJSU. And Instructure, when announcing its Canvas Network, was very clever in positioning itself as the friend of the university, in contrast to those shifty xMOOC characters. That makes sense for Instructure, given that the university is the company’s natural customer. For Pearson and other textbook companies, their natural customer is the instructor (even though the student pays). So it wasn’t terribly surprising to hear Sannier say, ”Teacher-led instruction is the future. But teacher-led instruction powered by much more powerful educational support technology and tools than in the past.” What is more interesting is how he threads the needle: ”Somebody will make a math class with 6 million students around the world. But it will be offered locally with teachers at a scale of between 1 to 20 and 1 to 50. Because teachers matter.”

This is where we start getting to some interesting frontiers, in my opinion. Leaving aside the exact teacher/student ratios, it seems likely to me that we will see a proliferation of different models that combine some elements of “massiveness” (or, at least, “scale”) with some elements of a small, possibly local class. Implicit in that mix is a new balance between the pedagogical work done by the course materials, the professor, and the students themselves. But how does the mix work? There is going to be more than one answer to that question, depending on the course subject, goals, and audience, but I’d like to start seeing some models.

Sannier provided some hints about how he sees it working. He talked about the course as a community, although it wasn’t clear to me how much he meant a community of students and how much he meant a community of teachers. Each course would have a range of modules. Teachers could select the modules that they think are best for their students and, presumably customize. He suggested that analytics could “maintain individuality while providing the benefits of scale.” I suppose what he means by that is that analytics can enable adaptivity (of both the content and the teacher) to personalize to the student’s needs while, on the other hand, identify the best elements of the course and provide direction for improvements to the course design by analyzing the patterns in large numbers of students using the courseware.

Another way in which he talks about the benefits of scale is in the course design itself. He said there simply aren’t resources to design cutting-edge digitally enhanced courses at local institutions. But if you amortize the costs over millions of students, you can bring substantial resources to bear even at a reasonable price to students. This is one area in which Pearson has a big leg up on the MOOCs. They are struggling to figure out how to charge any money at all for their products. In contrast, a textbook publisher can bring a $50 product to the market and have it look cheap. That means Pearson can make substantial investments in course development and be fairly confident that they will get a return on their investment.

All of this is very interesting, but still vague. What, exactly, would a course design look like? What kind of resources would it include? Adrian mentioned the specific example of a course called Habitable Worlds, developed by Professor Ariel Anbar of ASU and soon to be brought to the OpenClass platform. I asked for and received permission to talk to Professor Anbar about his course, and will write about that conversation in a future post.

The post OpenClass vs the MOOC appeared first on e-Literate.

The transformation of the higher education LMS market continues, and I expect more changes over the next 2 – 3 years. However, it seems time to capture a key segment of the market based on two recent announcements that directly impact large online programs.

UMUC Selects Desire2Learn

Following a multi-year strategic study and product selection, the University of Maryland University College (UMUC) selected Desire2Learn to replace the homegrown WebTycho LMS in use for over a decade. While there has been no press release, there are now official documents publicly available that describe this effort. Per the Faculty Advisory Committee (FAC) Newsletter:

As announced last year, UMUC is planning to replace WebTycho with another learning management system. After a great deal of questions for the vendors, references checks and tests of functionality, Desire2Learn was chosen. More than 200 faculty, staff and students worldwide were part of the decision which was reported to have been by an overwhelming majority. The roll out will occur gradually over the next year or two. [snip]

Not only is UMUC moving away from the WebTycho platform to the Desire2Learn (D2L) learning management system, but additional changes are in the works which will transform the online learning experience for our students and faculty.

UMUC is part of the University of Maryland system, and it has a global focus – serving over 90,000 students worldwide.

Mississippi Community Colleges Select Canvas

The Mississippi Virtual Community College (MSVCC) provides the LMS for the system’s online program as well as for all 15 campuses. After a 6-month RFP process, MSVCC will be migrating from Blackboard Learn and Desire2Learn and implementing Canvas as the new LMS. From the press release:

The Mississippi Community College Board and the state’s 15 community colleges recently selected Canvas by Instructure as the learning platform for the Mississippi Virtual Community College (MSVCC). This partnership marks the beginning of a five-year engagement with Instructure, an innovative company in eLearning technology. Canvas will provide the critical tools necessary to ensure that the MSVCC accomplishes its strategic goals centered around teaching and learning. Because Canvas runs in “the cloud,” MSVCC will recognize a cost savings of approximately $1.5 million per year. The MSVCC is expected to “go live” on the Canvas platform beginning with the summer 2013 term. [snip]

The MSVCC is a collaborative effort among all 15 community colleges to offer the full array of academic, career, and technical courses necessary to earn an Associate of Arts or an Associate of Applied Sciences online. The MSVCC dates back to fall 2001 and is recognized nationally as among the best eLearning models in the nation. During the initial launch of the MSVCC, a total of 4,781 students were enrolled in 8,281 classes. During the fall 2012 semester, 28,576 students took 60,883 classes.

Disclosure: I have advised on both UMUC and Mississippi projects as well as several additional schools listed below. All information for this post, however, is based on public information unrelated to the consulting projects.

Growing Role of Non-Profits in Online Education

Both of these selections highlight the growing role of non-profit institutions – both public and private – in online education. Throughout the 2000s, for-profit institutions such as the University of Phoenix and DeVry provided the bulk of large online programs, with only a handful of non-profits providing online programs at scale. In the past two years, however, traditional non-profit programs have grown significantly while most for-profit schools have shrunk in enrollment.

In the following graphic, the size of the bubble represents enrollment in online programs or courses. For example, the University of Phoenix has almost 300,000 online students, while UCF has approximately 30,000 online students. Bubble sizes are estimates only, as there is no standardized method for accredited institutions to report online enrollment. The source of enrollment numbers is a combination of IPEDS data, web site research, press releases, and secondary studies such as Parthenon Group’s report.

A few notes:

• One major difference between large online programs and the broader higher education market is the strong usage of homegrown systems in online. Michael and I have written about the University of Phoenix and their investment in learning platforms, but Rio Salado, WGU and ITT also use homegrown systems.
• The online program market is where Pearson eCollege (aka LearningStudio) does most of its business. The commonly-cited numbers from the Campus Computing Project – that eCollege has a 1% market share – does not include for-profit schools and is based on number of institutions. By looking at enrollments, it is clear that Pearson is one of the biggest players in this segment of the market.
• Like Pearson, LoudCloud Systems has focused on the online market, particularly with for-profit institutions.
• Blackboard faced a dangerous situation with the end-of-life of WebCT and ANGEL product lines (with the ANGEL line subsequently revived). As seen here, Blackboard has successfully moved many of their large online customers into Blackboard Learn – including UMassOnline and SUNY Learning Network. Blackboard still has online schools (e.g. Excelsior) facing an LMS decisions due to perceived or real end-of-life of LMS products.
• Desire2Learn has built up a niche of public online programs, particularly with the UMUC selection, added to the Colorado Community College Online program.
• While Canvas has based its growth mostly on traditional face-to-face programs, the additions of University of Central Florida and MSVCC give Canvas its most significant presence for large online programs.
• American Public University is the only large online program in the US using Sakai.

The post Snapshot of LMS Market for Large Online Programs in the US appeared first on e-Literate.

I had an unexpected opportunity to chat with the Apollo Group’s Rob Wrubel last week. Rob is their Chief Innovation Officers and Executive Vice President. It was a short conversation—only fifteen minutes—but boy, was it dense with information.

For starters, I got more clarity on the $1 billion technology investment that we keep hearing about. That’s not just for their LMS. It’s basically their entire learning- and learner-focused technology portfolio. If you think about the strong shift to online education that’s been happening in the for-profit education sector and the fact that Apollo serves over 300,000 students, it makes sense that they would need to make a massive investment in modernizing their IT infrastructure, including the LMS, but also their registrar software, their student/customer tracking software, their data centers—pretty much everything from soup to nuts. A billion dollars is still an impressive amount of money for a company to invest in just about anything, but to put it into perspective, Apollo makes about $4 billion in annual revenues. So an investment of 25% of that over a few years to upgrade their entire mission-critical IT infrastructure sounds forward-looking but reasonable.

We also talked a little bit about Apollo’s learning analytics. Wrubel identified a number of different layers. The first layer is their persistence and retention early warning system. This is the kind of learning analytics that is most widely adopted and well understood. (I have written about Purdue’s Course Signals as an example of this kind of system.) The second layer is cohort matching. Apollo places students into twelve- to eighteen-person learning cohorts. As you might imagine, getting the right mix of students in terms of their skills and abilities can be critical to the success of the cohort. The company has invested in analytics that help them get the right groups together. (By the way, this is one frontier that I think the MOOCs are going to eventually going to have to tackle. I suspect that one reason Coursera’s peer review functionality has gotten panned, despite a growing body of literature that calibrated peer review can work, is that they have vastly more heterogeneous groups than a traditional university class setting.)

The third layer is tracking student progress across the curriculum, looking for the drop-off points. In a way, this is a complement to the retention early warning analytics, but looking at it from a perspective of finding the rough patches that are likely to cause students trouble and sanding them down, as opposed to finding students who are in rough patches and helping them through. Wrubel put a lot of emphasis on completion, and specifically contrasted that emphasis to the high drop-out rate that we see in MOOCs. He also talked particularly about what he called “foundational learners,” which I suppose is a euphemism for remedial learners. ”Foundational learners just carry more risk factors coming into the risk factors,” he said. “We have done a lot in this area.” He talked about providing many small doses of remediation over time, as opposed to pulling a student out for an eight-week block of remediation.

Next, he talked about using the technologies and approaches of Carnegie Learning to improve outcomes. I have written previously about the powerful techniques being developed at Carnegie Mellon University and the University of Pittsburgh to identify skill ladders in learning a particular subject and remediating students as they work their way up those skill ladders. Apollo acquired Carnegie Learning, a commercial spin-off that develops courseware based on this approach. The education giant clearly has big plans for leveraging their acquisition. Wrubel really emphasized the value that Carnegie brings to the table. He scoffed at using Google-like tricks to personalize learning through big data magic (which is very much in line with my recent critique). Instead, he talked about automating the thus-far labor-intensive process of discovering skill maps for different subjects and disciplines. That’s a pretty ambitions and important research project, and I will be curious to see what they are able to accomplish.

Finally, he talked a little bit about that activity stream work (and patent) that Phil wrote about recently. We were at the very end of our time, so I didn’t get as much on this as I would have liked, but the gist is that the innovation is not so much on the idea of an activity stream as it is figuring out which bits of activity stream data are important to which stakeholders. What do the students need to see? How about the teachers? How about the analytics systems? At this point I’m extrapolating (and speculating) from a few remarks, but my sense is that the technology may be more properly thought of as a data bus than as a Facebook-like interface. “Activity stream” should be thought of in this context as the underlying data structures that are being routed, filtered and prioritized, but it’s the verbs in the sentence rather than the nouns that Apollo seems to be emphasizing.

I am hoping to have some follow-up conversations with Rob and other Apollo stakeholders and will let you know what I learn.

The post Apollo Group’s Technology Investments appeared first on e-Literate.