The TIEE Research Practitioners Project: Faculty Investigating Active Teaching and Student Learning

Introduction

Interest in pedagogical scholarship conducted by faculty themselves is growing and becoming an increasingly accepted form of inquiry in higher education. Huber and Hutchings (2005) trace the development of this phenomenon from Boyer's (1990) definition of teaching as scholarship to the establishment in 1998 of the Carnegie Academy for the Scholarship of Teaching and Learning. More recently, the extensive pedagogical work being done by faculty in the sciences has been recognized (e.g. National Research Council 2002, Fox and Hackerman 2003), for its potential to drive change in science education.

The scholarship of teaching and learning incorporates a number of formal and informal modes of inquiry. Action research is done by practitioners (teachers, student services staff, and others directly responsible for creating learning opportunities) who ask questions and design studies that will directly inform their practice (Stenhouse 1975). Though done infrequently in higher education, it has been successfully employed as a model for faculty professional development that emphasizes personal improvement through structured inquiry (Zuber-Skerritt 1992). Classroom assessment (Angelo and Cross 1993) and classroom research (Cross and Steadman 1996) are increasingly popular methods for college faculty to engage in ongoing, practical inquiry into student learning. Handelsman et al.'s (2004) paper in Science uses scientific teaching to describe teaching that is based on theories of teaching and learning and is informed by empirical evidence. In this paper, the term practitioner research will be used to describe inquiry conducted by faculty on their own practice, encompassing action research, classroom research, and scientific teaching.

Weimer (2006) summarizes what is known about practitioner research and provides some compelling reasons why faculty should engage in this type of inquiry: it provides a stimulus for thinking more deeply about teaching and learning, helps faculty acquire knowledge they can use to improve practice, and provides a means to reinvigorate collegial discussions about teaching. Practitioner research enables a different kind of learning than a faculty member typically gets from attending workshops or reading about teaching innovations. It involves the development of knowledge through critical reflection on our own actions and observations (Schon 1995) and may even require ongoing conversations with colleagues to help clarify interpretations of data and events (Brookfield 1995). Practitioner research allows faculty to focus on problems specific to their disciplines and student populations. In ecology education, for example, conducting research in their own classrooms should help faculty learn more about what limits their students' understanding of challenging and important ecological concepts or ways of thinking as well as how they can monitor change in student learning by focusing on specific topics of particular interest to them (Bransford, et al. 1999).

Many outstanding professional development opportunities are available to ecology faculty to assist them in developing the skills needed to pursue this type of research, such as the Faculty Institutes for Reforming Science Teaching (FIRST II 2004); however, the number of faculty actually conducting practitioner research appears to be relatively small. An investigation of faculty use of active, student-focused strategies using materials published in Teaching Issues and Experiments in Ecology (TIEE) (D'Avanzo, et al.. 2006) found that, while around half of TIEE users considered themselves very familiar with non-traditional pedagogies, and the majority developed their own adaptations of the materials or used them as a model for innovation, very few were systematically studying the impact of their innovations on student learning. One continuing challenge, and one which TIEE was designed to address, is that practitioner research is still not considered as valid as more familiar modes of scholarship by many faculty and academic departments (Weimer 2006).

The TIEE Research Practitioners Project was initiated to address these issues, with a goal of supporting college biology and ecology teachers in their efforts to formally investigate the effectiveness of new instructional strategies, in order to expand their own knowledge base about what works as well as to gain scholarly recognition for their work. The Project was designed to investigate whether a collaborative team with similar research interests, and with the guidance and support of experienced practitioner-researchers, could achieve significant, publishable results even in a single academic year. Specific research questions that guided this study included:

1. How do faculty conceptualize, design, and conduct research on their own teaching?
2. What do faculty learn as a result of doing research on their teaching, and how does this impact their practice?
3. Will practitioner-research conducted as part of this project be recognized as scholarship for tenure and promotion purposes at the faculty member's institution?

Methods

Recruitment and selection of participants

Early in 2005, participants for the TIEE Research Practitioner Project were recruited using mailing lists of TIEE users and ESA Education Section members. The call for participants and application instructions were posted on the TIEE website. Out of a sizeable applicant pool, fifteen faculty were selected to participate based on their commitment to the project and to non-traditional teaching approaches. They include five faculty from state colleges and universities, nine from private colleges, and one from a community college. Four participants were new to their institutions and would be up for tenure in the near future; three of these had relatively little teaching experience. Of the remaining eleven, all had extensive teaching experience, but widely varied levels of experience with student-active teaching methods. Only one had previously published work related to ecology education.

Participant activities

Participants agreed to take part in the following activities:

In addition, the majority (10) of the participants further developed their research into the articles in the present volume of TIEE.

August 2005 workshop

At the first TIEE Research Practitioner workshop, participants engaged in activities designed to help them choose a research focus, develop an initial research design, and choose assessment methods. The participants formed themselves into teams, each addressing a learning outcome important in science and/or environmental education: data analysis and graphing, experimental design, conceptual understanding, and attitudes and values. Members of each team designed studies to focus on a shared research question, such as, How can students' quantitative skills be improved? and Can classroom learning (increased knowledge) lead to changes in environmental values? Team members developed research plans and shared these electronically in the weeks following the workshop.

Team collaborations

To facilitate ongoing collaboration during the academic year, the teams used a telephone conferencing system and an email list provided by ESA. The conferencing system allowed team members to join in on scheduled conference calls using a toll-free number. Conference calls were scheduled about every month for each team during Fall 2005 and Spring 2006. The email list was used to distribute plans, assessment instruments, resource materials, and draft reports as work progressed.

Posters and papers

Twelve participants presented their research projects in the education poster session at the 2006 ESA meeting. Four members of the data analysis and graphing team collaborated on their study and presented a poster together as a team. The other three participants decided that they had not made enough progress with their research by the poster submission deadline in March 2006, and they plan to present posters at the 2007 ESA meeting. Ten participants expanded upon their work in their posters with full research papers, published in this special volume of TIEE. We regard this as a first step towards eventual publication in a more traditional education journal.

Assessing the participants' progress and outcomes

To determine whether the participants were successful in achieving the Project's goal of successfully conducting research on their own teaching and increasing their knowledge base about the impact of active teaching methods on student learning, the Project evaluator (Morris) conducted a descriptive study that examined patterns observed in the work, expressed beliefs, and self-reported actions of the participants. As a multi-method, qualitative program evaluation, the emphasis is on creating a detailed picture of what the participants accomplished and what they learned from the process. It is not explanatory in the sense of attempting to determine why participants made the decisions they did, or what factors in their background or experience affected their outcomes; more in-depth study of individuals or collaborative teams is needed to address these types of issues.

Research methods used included interviews, monitoring of the conference calls and email discussions, and analysis of participant work. Information was gathered from the participants on both a strategic schedule (mid-project and post-project) as well as at opportunistic intervals as communications occurred via conference calls and emails.

Thirteen of the fifteen participants were interviewed by phone as part of a mid-year progress check early in the spring of 2006. At the end of the project, ten were interviewed in person at the ESA meeting. All participants were interviewed at least once with interviews conducted in a semi-structured fashion, asking a similar set of questions but allowing the respondents to address events and issues they felt were important. Each interview lasted between 30 and 40 minutes. In preparation for the ESA poster session, participants submitted formal poster abstracts in March 2006 and shared drafts of their posters over the summer. Notes from the interviews and the contents of the posters were analyzed inductively for major emergent themes and are summarized in the Results and Discussion section without reference to the names of individuals.

Results and Discussion: What research practitioners learned and achieved

This section is organized into five parts, each addressing a different outcome of the overall project: 1) how the participants approached their practitioner-research studies and what they actually did during the period of the project; 2) how the participants conducted their investigations of student learning, including the methods they chose; 3) what the participants believed they achieved in terms of improving student learning; 4) what the participants believed they learned as a result of conducting a study on their own teaching and how it impacted their practice; and 5) the participants' views on whether their work would be accepted as scholarship. The other papers in this volume of TIEE provide more detail about each of the topics described below.

Implementing practitioner-research studies

Though almost all participants introduced published TIEE materials into their teaching as a basis for their research, most also modified TIEE materials, developed TIEE-like activities, and/or introduced teaching strategies consistent with inquiry-oriented science teaching, such as case studies. Each participant chose a focus question for their research that related to an issue of particular interest to them. Some questions focused on student skills of current concern in participants' departments, such as their ability to conduct scientific investigations or create and interpret graphs. Other questions reflected broader societal issues such as students' ability to understand their own decision-making processes when addressing ecological issues. Participants who did not actively or regularly collaborate with others during their studies still said that being part of collaborative group was motivating and useful, since they knew that other participants could provide help if needed.

Table 1 provides an overview of the participants' projects by research topic, institutional type, course type, and study design. The classes in which the participants conducted their studies ranged from small seminars to large lecture courses, and included both introductory and upper-division courses for both majors and non-majors. Almost all participants used published TIEE materials in their study courses, and several developed new materials that are potentially publishable in TIEE. Approximately half of the participants said they had used specific elements from TIEE activities or had modified existing TIEE materials by adding additional literature, data, and images. This is consistent with previous research on TIEE (D'Avanzo, et al. 2006) that indicates most users do not use TIEE materials as is, but modify them to meet their needs and interests. Several practitioners also used learning activities that are consistent with an inquiry-oriented approach to science teaching but do not necessarily fit one of the TIEE publication categories (Experiments, Data Sets, and Figure Sets). Such activities included case studies, writing assignments such as review articles, and term-long student-designed investigations.

Each participant focused their study on one category of learning outcomes: experimental design and research skills, data analysis and graphing, or environmental values (Table 1). The basis for these choices varied depending on the participants' current needs and interests. Some chose the focus for their study based on a special feature or goal of their course. One faculty member was teaching a newly designated quantitative emphasis course, part of her institution's effort to infuse quantitative skills across the curriculum. Others were concerned about some quality of their student population. Those with large numbers of career-oriented non-majors (such as pre-medical students) wanted to improve their students' attitudes toward the study of ecology and help them clarify their views on environmental issues. Others who were teaching courses for majors felt it was important to focus on experimental design, since this is such an important outcome of an undergraduate program. One decided to conduct her study in a manner that would inform her department about whether or not majors were getting sufficient preparation in experimental design in their introductory courses.

A major feature of this project was the formation of collaborative teams that worked together on classroom research projects. Most participants commented that they had really benefited from their colleagues' ideas that were shared during the conference calls and through postings on the project email list. The data analysis/graphing team, in particular, maintained a high level of collaboration throughout the project, with most members participating in scheduled conference calls, sharing their research plans and assessment instruments over email, and scheduling their own conference calls while planning their team poster. Members of this team who did not get their studies started in fall were able to use work done by the other members in the fall to jump-start them in the spring.

All of the participants said they probably would not have gotten involved in education research without the opportunity to work with a collaborative group. Several said that, even though they had not participated in the conference calls and email discussions as often as they would have liked, just knowing that others were out there working on the same issues was motivating, plus they knew they could get help and support if needed.

Designing and conducting studies on student learning

Participants' research methods emphasized direct measures of student learning, including pre- and post-tests, class activities, and projects. Indirect measures such as surveys and reflective writing were used to supplement and expand upon the direct measures. Participants who included qualitative data in their studies had extensive discussions about methods for analyzing these types of data and the validity of various analytical approaches. Overall, participants exhibited a high degree of concern for the appropriateness of their data collection and analysis methods.

All of the participants identified and/or developed instruments to be used as pre- and post-tests to gather data on the impact of the new teaching strategies they used. There was much discussion at the beginning of the project regarding whether a pre- and post-test needed to be identical and also whether a pre-test was necessary in all circumstances. All participants decided to administer some kind of pre-test, but this was not always identical to the post-test. Some participants embedded sub-sets of the pre-test questions into exams which then served as post-tests, while others used a pre- and post-test that addressed the same concepts and skills but used different questions. In order to assess students’ understanding of graphs of different types and levels of complexity, members of the data analysis/graphing team designed their instruments so that the pre- and post-test items used the same type of graph but were not the same exact graph. As several of these team members commented, this required them to be very clear about what data analysis skills they wanted to assess.

In designing pre- and post-tests, participants used published sources as well as items they developed themselves. Two of the teams collaborated extensively on their choice of pre-post assessment questions. The data analysis/graphing team worked together to develop test items that reflected their desired outcomes while the attitudes/values team also used an existing published instrument they used as is. Pre- and post-tests developed by the participants tended to be a mix of multiple-choice and open-ended questions. One participant, who investigated students' understanding of the scientific process, used as her pre-test an essay that the students wrote regarding the relationship of hypotheses, facts, theories, and predictions. Her post-test included some similar short-answer questions as well as multiple-choice questions that required students to evaluate specific examples of scientific investigations.

The initial ESA workshop emphasized using other means of data collection in addition to the pre- and post-tests throughout a semester, and almost all of the participants implemented such assessment strategies. Many used new or existing teaching strategies as sources of assessment data, such as case studies, lab and field investigations, review article writing, and in-class TIEE activities. Several who used TIEE activities or other in-class learning activities used non-graded written questions to assess students' progress. At least four participants used reflective writing or classroom assessment strategies such as the one-minute paper, either throughout the semester or to supplement a post-test. There was much discussion on how best to analyze and organize data from these sources for inclusion in published research, and all participants reported becoming more confident in working with qualitative data, particularly in view of its value in helping explain and triangulate with their quantitative pre-post test data.

Impact on student learning

Participants were able to use their research results to describe significant improvements in their students' learning, although several said they were still dissatisfied with students' progress. Some participants found that their assessment methods needed improvement in order to effectively characterize student learning and measure learning gains. Participants encountered little or no student resistance to the novel instructional and assessment methods they implemented as part of their research projects, with most reporting that students really enjoyed the new activities and felt they had learned from them.

All of the participants felt their students had learned more as a result of their new approaches. Three of the experimental design team members saw significant increases in their students' abilities, noting that students now saw research as iterative, not linear, were able to pose their own hypotheses and design their own experiments, or were much better at analyzing strengths and weaknesses in published research. Members of the attitudes/values team also saw improvements in their students' critical thinking skills, including their ability to analyze issues. One participant's students gave presentations on a current issue, its underlying science, and related policy. The participant invited her colleagues to the presentations, and they found the students' work to be of very high quality. Additionally, students' course evaluations were very positive, and after the class students reported that they had been able to use what they had learned in class in other areas of their lives.

Although the data analysis/graphing team felt they had figured out ways to teach certain skills or explain certain concepts so that the students finally got it, the team had other items on their agenda with which they were less successful. One team member focused on the meaning of p value, using a variety of activities throughout the term, but never felt that her students really understood it. Other members of this team believed their students had really improved their abilities to create graphs but found that students still had problems with interpretation, while others found just the opposite.

Most participants felt that they needed to improve their assessment methods in order to get better data. Some felt that their test questions did not adequately target the specific skills they were trying to assess. One participant used graded tests as a primary source of data, but the test items got more sophisticated throughout the term, producing what looked like declines in learning. She said she sometimes thought students learned more even when the data did not show it. Another noted that simple questions did not provide good data, and that the better test questions were those that provided a scenario or required application of concepts. Scenario-based questions provided more information, such as showing that students could understand and use concepts even if they could not use correct terminology or scientific conventions.

Contrary to the experience of many faculty who introduce new teaching and assessment strategies, the participants in this project encountered little resistance to their new methods from students. They noted that their students really enjoyed conducting investigations in which there was no right answer, that they appreciated having control over their own learning, and that students who had previously been passive became much more engaged. One participant noted that her class became more focused and that its organization and flow were improved. This participant and one other said their teaching evaluations improved significantly over the previous year, and both believed this was due to the active learning approaches they had implemented.

Impact on faculty learning and teaching practice

All participants reported an increased understanding of student learning and what works. All participants reported making significant changes in their teaching practice, with their assessment data informing their ongoing decision-making processes. The most frequently cited lesson learned among participants was the importance of being extremely clear about what learning outcomes were being addressed in order to effectively help students learn and demonstrate those learning outcomes.

A major question of this study is whether the participants made significant changes in their teaching practice based on their use of TIEE materials and their investigations of student learning. All fifteen of the research participants said that they gained major insights into student learning and found new and more effective ways to improve it as a direct result of participating in this project.

Table 2 categorizes and provides examples of some major changes participants made in their teaching practice. These fall into two areas that are consistent with most major recommendations for improving science teaching: moving from a content-centered to a student-centered approach, and engaging students in doing real science.

Several participants talked about their switch from a content-centered approach to a more student-centered approach. One said she realized that she can sacrifice time previously devoted purely to content; her approach had been, just lecture, lecture, lecture, but her assessment plan made her stop to do the assessments and involve the students. Another said he had previously started with theory and moved to issues and applications later; now he interweaves the two. He felt that his research plan kept him focused on issues and helped him avoid going off on a theory tangent that may have been interesting to him but not useful for the students. Two other participants used classroom assessment strategies extensively to help them and their students understand their learning difficulties and develop interventions. One said she has learned so much about students’ assumptions and level of knowledge that this has helped her to completely change her approach to teaching.

Most of the participants said they were able to give students an increasing amount of freedom to ask their own questions and pursue their own investigations. One said he was allowing students to be more independent in all aspects of the course. Three said they were enabling students to choose their own problems for investigation, pose their own hypotheses, and work with their own data. One of these said that he had really moved away from directive, cookbook approaches to labs, and that giving students raw data and asking them to figure out how to display and interpret it was a major change for him. Others designed their activities to initially provide students with a high degree of structure and support, but then gradually required more self-direction, ending with open-ended, student-directed projects. Participants found this to be useful primarily for teaching scientific investigation across a semester but also in collaborative, problem-based activities in class, and they felt that it really improved students' abilities to become more self-directed.

All of the participants said they had learned skills and developed tools needed to improve their teaching. Many commented on the value of practitioner research to support decision-making and said that having more systematic data informed their teaching or at least confirmed that they were heading in the right direction. One said she had always believed that systematic approaches to teaching were for those who had less native ability, but now feels that this is something everyone should pursue. Many participants commented specifically on the importance of being clear about what they were teaching and assessing. Most said they were able to ask better questions as a result, while others felt they needed more work in this area. One said her efforts to improve how she taught students to do research required her to deconstruct this entire process; another who focused on cutting content said she had to back up and really think about what she wanted her students to learn.

Perceived value of this research as scholarship

Most participants plan to continue or expand their studies with the goal of further publication. Only three of the participants believe their work will contribute favorably to promotion and tenure reviews by their department; however, others will use their work to fulfill other professional responsibilities such as improving teaching practice. Many participants were able to clarify their views regarding the value of practitioner research and the qualities necessary for it to be considered scholarship.

All but one participant said they planned to continue the research they initiated this year and publish in other venues. Ten have already made progress toward this goal with the papers published in this volume of TIEE. Three participants explicitly stated that they believe their work will count in upcoming promotion and tenure reviews, but even among these, they were uncertain about the extent to which the work will be considered the equivalent of traditional disciplinary research. One of the three said that her colleagues had initially been supportive but now are really behind her and see the value of this kind of research, while another said that his project would be a major component of his scholarship in his upcoming review. The third is up for tenure next year, and she will use her project to provide the required documented evidence of teaching success, not as published research. Two participants said that they were sure that their research would not be considered in promotion/tenure reviews; one said her study would not count in her upcoming tenure review, since at her institution, excellent teaching is a given.

Almost all the participants said they were very interested in sharing their work with their department simply to facilitate focused conversations about teaching and student learning, regardless of whether it would count. Three participants plan to connect their work to their institution's plan for departmental assessment of student learning outcomes; one of these says he expects to be asked to take a leadership role in program and institutional assessment planning for an upcoming accreditation visit. Two participants are also planning an extended study of student progress in lab classes versus non-lab classes to help make the case for the value of labs in achieving key learning outcomes for majors.

Several of the participants had valuable insights into the nature of the kind of classroom research they were conducting and its relationship to more familiar kinds of scholarship. One said he initially had some hesitations about the validity of the kinds of data and data analysis that were being advocated, but found that you really can learn something useful. As he said, it is a different kind of research than what scientists normally practice, and it addresses different kinds of questions. He now has lots of ideas about how to extend his inquiry into these questions. Another participant noted that the most worthwhile thing has been standardizing faculty observations regarding key student outcomes such as graphing skills and experimental design. As he put it, faculty assess student learning all the time, but in a highly contextual and anecdotal manner; in this project, the participants were able to make observations in such a manner that consistent inferences could be drawn from them.

Several participants also noted that it was often uncomfortable starting an experiment in an area outside their normal areas of expertise. Not being professional science education researchers and part of a research tradition made it harder to predict what would be a productive line of inquiry; however, others, particularly those who had some experience with informal classroom assessment, found that the project helped them formalize their inquiry and gain a better understanding of what they could do to help students learn.

Conclusions

The TIEE Research Practitioners Project was effective in enabling its participants to formally investigate changes in their teaching practices and gain knowledge and skills that informed their teaching decisions. In almost all cases, the practitioner research component of this project, which provided an opportunity to formally investigate student learning, provided not only the motivation but the knowledge and support needed to implement and maintain changes in teaching practice. The project provides additional evidence that practitioner research is a viable model for faculty-driven improvement of teaching and learning.

Though forms of practitioner research such as classroom assessment are gaining ground in many colleges and universities, it does not yet represent a widely accepted or widely known approach to improving college teaching. With the pressure of other priorities and the traditionally solitary nature of most college teaching, practitioner research will continue to emerge only in isolated pockets. Higher education institutions need to consider how they can help overcome these obstacles. Faculty members who are less experienced with this type of inquiry need guidance in the process of investigating student learning and using that information to modify their teaching practices. As faculty gain experience, they must engage actively in discussions of quality in practitioner research while pursuing models and approaches such as peer review that will enable them to conduct high-quality studies. Finally, academic departments and administrators in all types of institutions, not just research intensive universities, must recognize the value of this type of scholarly work and heed recommendations for elevating the scholarship of teaching and learning.

More widespread use of practitioner research will enable further study of its long-term impact on faculty teaching practice and student learning. What factors will encourage faculty to continue investigating student learning over time? Is it necessary for faculty to continually transform their teaching, or will they eventually arrive at “best practices” that no longer require major change? Or, will the processes of practitioner research themselves be what allows us to maintain a connection with our ever-changing student population and engage them in meaningful exploration of their own learning? While we have growing evidence that practitioner research is valuable for individual faculty, its meaning within, and potential impact on, our system of higher education as a whole is yet to be explored.

References

• Angelo, T. A., and K. P. Cross. 1993. Classroom assessment techniques: A handbook for college teachers. Second edition. Jossey-Bass, San Francisco, CA.
• Boyer, E. 1990. Scholarship reconsidered: Priorities of the professoriate. Carnegie Foundation for the Advancement of Teaching, Princeton, NJ.
• Brookfield, S. 1995. Becoming a critically reflective teacher. Jossey-Bass, San Francisco, CA.
• Bransford, J. D., A. L. Brown, and R. R. Cocking. 1999. How people learn: brain, mind, experience, and school. National Academy Press, Washington, DC.
• Cross, K. P. and M. H. Steadman. 1996. Classroom research: Implementing the scholarship of teaching. Jossey-Bass, San Francisco, CA.
• D'Avanzo, C., B. W. Grant, D. Morris, S. Musante, J. Taylor, J. Riney, and D. Udovic. 2006. Design and evaluation of TIEE, a peer-reviewed electronic teaching resource. Frontiers in Ecology and the Environment 4: 189-195.
• FIRST II (Faculty Institutes for Reforming Science Teaching). 2004. http://www.first2.org.
• Fox, M. A. and N. Hackerman, editors. 2003. Evaluating and improving undergraduate teaching in science, technology, engineering, and mathematics. National Academy Press, Washington, DC.
• Handelsman, J. D. Ebert-May, R. Beichner, P. Bruns, A. Chang, R. DeHaan, J. Gentile, S. Lauffer, J. Stewart, S. M. Tilghman, and W. B. Wood. 2004. Scientific teaching. Science. 304: 521-522.
• Huber, M. T. and P. Hutchings. 2005. The advancement of learning: Building the teaching commons. Jossey-Bass, San Francisco, CA.
• National Research Council. 2002. Scientific research in education. National Academy Press, Washington, DC.
• Schon, D. 1995. The new scholarship requires a new epistemology. Change. 27: 26-34.
• Stenhouse, L. 1975. Introduction to curriculum research and development. Heinemann Education, London.
• Weimer, M. 2006. Enhancing scholarly work on teaching and learning: Professional literature that makes a difference. Jossey-Bass, San Francisco, CA.
• Zuber-Skerrit, O. 1992. Professional development in higher education: A theoretical framework for action research. Kogan Page, London.