The Emperor’s New Clothes: 4 Steps to Successful ILS Implementation
Copyright 2003 Miguel Guhlin

“Sometimes,” the tech director I chatted with at a recent conference shared with me, “I feel like I’m one of the adults in the Emperor’s New Clothes book (Read it online). I just found out we’re going to implement an ILS at the middle school level.” This comment brought to my attention a disturbing shift in the approaches technologists are being forced to take to prepare students for No Child Left Behind accountability measures.

As Richard Elmore points out in his article in Educational Leadership (November, 2003), "When we bear down on testing without the reciprocal supply of capacity. . .schools search for short-term solutions--test preparation--rather than longer-term, more powerful solutions, such as curriculum-focused professional development." It is this push that drives some central office administrators in public schools to turn to a sleek, short-term success solution like integrated learning systems (ILS).
ILS Selection Process 1. Form a team comprised of the participants—teachers, community members, administrators, students (Hill, 1993). 2. Have teachers select the software and ILS. Once selected, the system should be integrated into the local curriculum by the appropriate district curriculum staff. This role should not be usurped by computer teachers or the district computer coordinator or director (Wiberg, March 1992). 3. Using your district curriculum as a baseline, look at the ILS curriculum to see how it matches your curriculum (Mageau, 1992). 4. Use the rubric available at http://www.mguhlin.net/portfolio/writings/2002/ilses.html
Many articles regarding integrated learning system alternate between praise and disdain. The fact is while integrated learning systems hold great potential for impacting test scores over the short-term—such as a year--in the long term they are not as effective. But, for central office, short-term gains in student achievement are worth the $50,000 per ILS campus implementation, not to mention the cost of a networkable computer lab with district aggregation capabilities (an estimated $20,000).
Campus administrators and teachers find themselves in an awkward situation—they know the research does not support integrated learning systems, but they are forced to implement a solution that just does not work well for most students (Becker, 1992). Some other reasons why ILSes do not prepare students for a world that requires ill-structured problem-solving:
  • The ILS won’t work if you use it for less than 3 times a week and just leave it to “get on with things on its own.” (Ian Hedley, Carter Community School, ILS Coordinator).
  • ILS successful results—often self-reported or by a subsidized researcher--relate to how they were publicized. Studies in reports from the vendor all show substantial positive effect sizes. Those from independent sources show modest or negligible effects (Bracey, 1991).
  • The ILS can teach routine skills but they cannot teacher higher order thinking skills or conceptual thinking. Students using an ILS for more than a term had become less enthusiastic about the system (White, 1993).

As a bilingual teacher who witnessed integrated learning systems and their lack of efficacy in the famous Edgewood ISD, I am alarmed at how often students in at-risk situations are forced to use integrated learning systems of some sort. It is frightening to consider a return to these methods of “improving” student achievement. As it states in the book Towards Digital Equity: Bridging the Divide in Education, economically disadvantaged students “learn to do what the computer tells them, while affluent students, learn to tell the computer what to do” (Solomon, Allen, Resta, 2003). It is clear that my children will learn to tell the computer what to do, while students in ILS-using school districts will be told what to do, perpetuating the cycle of passivity and lower-order thinking.

IMPLEMENTING TOP-DOWN INITIATIVES SUCCESSFULLY
Despite Becker’s key research, some are still ignorant. Yet, what do you do if you are the technology director responsible for implementing an integrated learning system? Do you follow orders, remaining quiet, or resign in protest? This article does not address selection of ILSes—except as an aside—since implementation of ILSes is usually a top-down decision. Of course, you can always tell the emperor—represented by central office administration and/or school board—that his clothes are missing.

ILS BEST PRACTICES
As an administrator, it is your job to do what the superintendent and school board say. ILS research does show that ideal implementation can result in short-term gains. It is important that you know what to do. So, what are best practices for ILSes? This short article will get you on the right track. The information in this article is based on interviews and an intensive review of the research regarding integrated learning systems, only some of which is shared here. You can find the references online—with research excerpts—at http://www.mguhlin.net/portfolio/writings/2002/ilses.html

STEP 1: HIRE A COMPETENT, CAMPUS-LEVEL LAB MANAGER.
Integrated learning systems are often poorly implemented. Despite the best intentions of the company, and its professional development facilitators, top-down administrators are looking for a teacher-less, “thinking adult”-free lab environment. After a district spends so much money on an ILS, it seems nonsensical to spend more money on a lab manager and professional development. We know that computers cannot teach children to apply what they learn to real life problems (NCREL, 2002). Also, most teachers need a support person who is the in-house integrated learning system expert. These persons should be teachers who are knowledgeable about the curriculum and technical aspects of the system (Sherry, 1992).

In an interview with a campus administrator and lab manager where an ILS had been successfully implemented—that is, raised test scores for the first two years--the lab manager states that his responsibilities include the following on a daily basis. Note the percent of time spent each day on the responsibilities identified below:
÷ Technical Support and system administration (70%)
÷ Assisting teachers with implementation of the program (20%)
÷ Documenting and reporting progress (5%)
÷ Teaching students to use the ILS (5%)

According to the lab manager for this large school district campus, “The lab manager is critical to the implementation of the ILS. Teachers would not be able to do what I do. There is a little bit of technical background on how to manage the system. I can see a problem and trouble-shoot the issues.” He goes on to state that teachers’ first assumption. Common problems encountered include the following: 1) Computer might lock-up; 2) Computer will have problems with network settings; and 3) Lose icon/shortcut to the ILS and it has to be recreated. While professional development could be done for all teachers in an ILS implementation, a lab manager provides on the spot access to a technical problem-solver. Indeed, one might ask whether the lab manager was the student refining his/her higher order thinking skills.The lab manager also works to print out the reports and worksheets as requested by staff. S/he is able to provide necessary in-house support for teachers (Smith, 2002).

STEP 2: PROVIDE COMPREHENSIVE TRAINING FOR TEACHERS AND ADMINISTRATORS.
Hill (1993) states that it is important to provide comprehensive training for teachers and administrators on the ILS. This training should begin prior to its arrival and be done by the ILS consultants. In this way, Hill writes, the district can avoid the problem resulting from the lack of teacher support. This point in the research was supported by the interview with the campus administrator. He stated that “Prior to the implementation, we identified a focus population to be targeted based on interim testing and TAKS results. Teachers of those populations, the instructional coordinator and lab manager, were given one full day of training prior to the start of the program. Other staff were given a half day informational training.”

Key components of the training should include time for teachers to preview lessons. Teachers should be given additional in-school time specifically for previewing ILS lessons. The curriculum in an ILS is extensive and they cannot be taken home like other materials. A final suggestion is that teachers should have access to the ILS lab during after-school or weekend hours (Sherry, 1992).
STEP 3: GET TEACHERS TO GO TO THE LAB WITH THEIR STUDENTS AND CIRCULATE.
The benefits of having a teacher in the lab while students work on the ILS cannot be emphasized enough. Teachers are able to take advantage of “teachable moments,” relate how they are doing in the ILS curriculum to classroom learning. As the campus administrator shares, “Teachers can see where the students are at and try to evaluate what is going on.” This is supported by NCREL (1996) research that states that “Efforts must be made to facilitate students’ transfer of knowledge to other domains of experience.” Even though students may learn isolated skills and tools, it will be difficult for them to see how the various skills fit together to solve problems.

STEP 4: ENSURE THAT HARDWARE AND NETWORK IS IN PLACE.
No matter how wonderful the ILS curriculum, if the hardware is not in place, the $50,000 software investment is wasted. It is important that the hardware and network be able to support the ILS implementation. Hardware can be anything from routers/switches to the requisite headphones with microphone included. Without the former, connectivity is an impossibility. Without the latter, students cannot take advantage of the ILS’s multimedia capabilities.
While there are many issues to consider when selecting and implementing an Integrated Learning System—not the least of which is “Why abandon technology applications and problem-based learning for such a poor approach?”—these 4 points are critical to successful implementation. There are profound financial implications for each of these steps that should be considered PRIOR to undertaking an ILS implementation.

As Becker (1992) writes in his review of 100 ILS studies, there is little evidence of ILS impact on student achievement. Where differences were found between the achievement of ILS users and comparable non-users, Becker concluded they were too small to have any educational significance.
In this time of economic hardship, sticking to the principles we hold dear can be difficult. Perhaps, rather than remaining quiet, we can share the research with superintendents and principals that drill-n-kill has little place in the technology applications:TEKS classroom. Maybe then, the grown-ups will be unafraid to proclaim the emperor’s lack of clothing on behalf of the children we serve.


Integrated Learning System – Rubric for Selection
The following components need to be present in an Integrated Learning System. This information is based on a review of research related to Integrated Learning Systems, No Child Left Behind legislation. Refer to attached references.

An Integrated Learning System arranges an instructional program for the learner at the learner’s level and continuously updates the instruction according to a cumulative assessment of the learner’s progress. It also provides reports for administrators, teachers and parents that allow them assess students’ progress. Most ILS systems have ongoing expenses ranging from $10,000 and $30,000 annually per lab, in maintenance, license fees, software upgrades, as well as an investment in professional development that will equal the total expenditure on hardware and software (Mazyck, 2002).

||
Component
||
Exceeds Expectations

Meets Expectations

Unsatisfactory
||
||
Letter of Interest
||
Brief statement describing understanding the proposer’s understanding of the work to be performed
||
Firms’ process, technical experience, and industry experience in dealing with this type of work
||
Items that distinguish company from competitors
||
Detailed procedural specification as to how your organization intends to execute the requirements of RFP
||
Timeline for completion of the work.
||
Names/Resumes of staff members included in Vendor qualification
||
Company Information Form
||
Must be able to perform the following:
*
Curriculum includes reading/language arts, mathematics, ESL and bilingual education, grades K-8
*
Allows for management of data to provide specific learning objective deficiencies
*
Instructional Assignment and tracking
*
Standards aligned measurements
*
Test achievement forecasting
||
Time-phased cost analysis
||
Can export data and is ODBC compliant
||
Effectiveness of instructional technology for student achievement in the classroom, especially at-risk students and children of color. How did they address cultural diversity?
||
Compilation of knowledge on teaching and learning (learning styles, direct vs. indirect instruct, constructivism, multiple intelligences, etc). and its application to the product.
||
An understanding of the latest research in curriculum areas, such as math and reading, and a demonstrated application to products.
||
Third-party evaluation studies that include evidence of the solution’s effect on student achievement in school settings, recently as well as several years back.
||
What assessments work and how they link to curriculum and classroom instruction.
||
Professional development approaches and a detailed plan for classroom teachers on use of the ILS with students

Requirements


































Research Foundation




















||
Component
||
Exceeds Expectations

Meets Expectations

Unsatisfactory
||
||
Data collection methods
||
Data analysis methods and how the ILS will be adjusted to match our particular situation
||
Courseware contains dual language major content area components
||
Provides for management of learner’s level and continuous update of their instruction in line with a cumulative assessment of the learner’s progress
||
Allows participants to see where they are, what they are registered for, as well as how much they have completed in relation to their goals.
||
Needs to be focused on supportive cognitive requirements rather than focusing only on a behavioral model of learning
||
Diagnostic that obtains information regarding the students’ general ability level
||
Identifies previous experiences of student
||
Identifies students expectations of the instruction
||
Acknowledges students’ perceived relevance of the learning task
||
Provide support for varied ways for students to organize their knowledge
||
Provide students the chance for reflection
||
Consider individual learning styles
||
Facilitate knowledge transfer across contexts
||
Provide students with more responsibility for contributing to each other’s learning through cooperative learning components (positive interdependence).
||
Component
||
Exceeds Expectations

On Site Product Testing and Data Evaluation







Curriculum








































Meets Expectations

Unsatisfactory
||
||
Cooperative learning is an instructional strategy that is used to enhance the effectiveness of the ILS with diverse groups of students.
||
Provisions are made from professional development opportunities prior to deployment of ILS.
||
Professional development opportunities are available as follow-up sessions.
||
Professional development materials are available for teacher training.

Professional Development for Teachers
















Integrated Learning System
information is based on a review of research related to Integrated Learning Systems, No Child Left Behind legislation. Refer to attached references.

An Integrated Learning System arranges an instructional program for the learner at the learner’s level and continuously updates the instruction according to a cumulative assessment of the learner’s progress. It also provides reports for administrators, teachers and parents that allow them assess students’ progress. Most ILS systems have ongoing expenses ranging from $10,000 and $30,000 annually per lab, in maintenance, license fees, software upgrades, as well as an investment in professional development that will equal the total expenditure on hardware and software (Mazyck, 2002).


||
Note that there is an extensive rubric for evaluating Integrating Learning Systems online at http://www.nald.ca/fulltext/pat/anaBC/page89.htm

This is the culminating work of British Columbia’s assessment of multiple ILSes, including AutoSkill, Successmaker, Jostens, Pathfinder, and Plato.
||


Research Findings on ILS Implementation:


#
Teacher who is more involved with the ILS work of her students will likely produce the greatest learning gains
#
Teacher who had far and away the most successful effect size [substantial increase in test scores] knew the most about the system, knew the most about what the kids were doing in the lab, and went back to the classroom and made decisions about what to do based on that information (Mageau, 1992).. When ILSes are uses as adjunct activities or when teachers don’t chart progress and use results to complement curricular efforts, they have no positive effect on student achievement or attitudes (NCREL, 2002).
>
#
Get teachers to go to the lab with their students and circulate among them while they’re online. (Mageau, 1992).
#
Use cooperative learning approaches to enhance the effectiveness of ILSs with diverse groups of students (Becker, 1992a; Brush, 1997a; Brush, 1997b; Brush, 1998; Hooper & Hannafin, 1991; Hooper, Temiyakam & Williams, 1993;Mevarech, 1994 as cited in Mazyck, 2002). Cooperative learning paired with ILSs produces more willingness to take on difficult tasks and persist, long term retention of what is learned, higher level reasoning and metacognitive thought, creative thinking, transfer of learning, positive attitudes towards the task being completed and greater time on task. It also improves student achievement and attitudes in computer-based environments.
>
#
When pairs of students work cooperatively to complete exercises in an ILS, they OUTPERFORM their counterparts who use the system on an individual basis (NCREL, 2002).
#
Effort must be made to facilitate students’ transfer of knowledge to other domains of experience (NCREL, 1996). “Students may learn isolated skills and tools but they will still lack an understanding of how those various skills fit together to solve problems and complete tasks.”
#
Should be an integral part of the district’s educational strategy and guide professional development of teachers (Tingey, 2002).
#
Comprehensive training for teachers and administrators on the ILS should begin PRIOR to its arrival and done by ILS consultants. In this way, the district can avoid the problem resulting from the lack of teacher support (Hill, 1993).
>
#
Plan site visits to talk about implementation, focus, funding, ongoing support and other relevant topics (Hill, 1993).
#
A team comprised of the participants—teachers, community members, administrators, students--should be in on the ILS selection (Hill, 1993).
>
#
Students may be off-task if they are not supervised in an ILS implementation (this research referred specifically to Successmaker). (Underwood et al, 2002).
#
Some conclusions on computer-based instruction (CBI):

##
The lower the grade level or ability of the students, the more effective CBI is
##
CBI is consistently more effective than traditional instruction but the amount of improvement is low to moderate
##
Structured CBI, with emphasis on direct instruction, is more effect in producing achievement gains than unstructured CBI
#
In a review of 100 studies of ILSes, Henry Jay Becker found that they “provided little evidence of ILS impact on student achievement. Where differences were found between the achievement of ILS users and comparable non-users, Becker concluded they were too small to have any educational significance.
>

Reading Research


#
Computer materials are most effective when they allow the learner some degree of decision-making and control over the task.
#
Interactive materials are imperative.
#
When possible, learners should be able to self-check by looking back at previous work or calling up help features.
#
Preference should be given to programs that require the reader to make decisions about the progression or direction of the content, thus, promoting active involvement.

References


Hill, M. (January, 1993). What’s new in ILSes? Electronic Learning, 12 (4), pg.12.

King, J. B. (December, 2000). Use of integrated learning systems to enhance the proficiency of poor readers in middle school. Unpublished manuscript.

Mageau, T. (January, 1992). Integrating an ILS: Two teaching methods that work. Electronic Learning, 11 (4), pg.16

Mazyck, M. (March/April, 2002). Integrated Learning systems and students of color: Two decades of use in K-12 education. TechTrends 46(2), pg.33

O’Neal, S. (June, 2002). The impact of ESEA’s scientifically based research requirement on schools’ technology solutions. T. H. E. Journal, pg.64

Tingey, B. & Thrall, A. (March/April, 2002). High stakes management. Multimedia Schools, 9(2), pg.S1-S7
Note: This article was written by the vice president of Successmaker.

Underwood, J., Cavendish, S., & Lawson, T. (2002) Integrated learning systems and educational outcomes. Available online at http://www.psychology.nottingham.ac.uk/research/credit/project/ils_evaluation/main.html
Prepared by Miguel Guhlin, mguhlin@yahoo.com
Director, Instructional Technology