SOJ Veterinary Sciences

Veterinary education like medical education consists of a spectrum of disciplines, fundamental sciences (biology, chemistry and physics), applied sciences (pharmacology, microbiology, immunology, physiology, anatomy) and clinical courses (application of the combined knowledge). The programme of learning adopted at Nottingham School of Veterinary Medicine and Science is one of integrated teaching to provide context specific delivery of fundamental sciences within body-system-based modules. This could lead to a potential fragmentation of certain core subject areas as they are integrated into modules. This shift in educational philosophy was designed to promote more independent learning by aligning fundamental topics to their final use in the clinical environment. The aim of this study was to investigate whether there are sound pedagogical reasons for this new curriculum. We report that undergraduate performance (in questions which tested different levels of knowledge) was not altered by module integration and that most undergraduates stated that they focused on the larger body-system-based modules rather than integrated modules contained within them. Thus suggesting that students are able to smoothly contextualise subject based disciplines within the larger systems-based modules. In conclusion, our study indicates that there are pedagogical reasons to continue with the integrated curriculum adopted at Nottingham.

Keywords: Pedagogy; Veterinary; Integration

As with other clinical/vocational courses, the veterinary curriculum, and its content, at Nottingham University are driven by an outcome based education system [1]. Decisions on the course content are driven by the required end point, learning outcomes based on the European Association of Establishments for Veterinary Education (EAEVE) and Royal College of Veterinary Surgeons (RCVS). As with a number of clinical sciences the breadth of these requirements has led to conflict between the volume of information delivered in clinical sciences and the time students can allocate outside their timetabled study to develop and learn the required concepts [2-3]. This has lead to a constraint on study time, leading to lower order learning as students use rote learning to cope with the large volume of factual information. The curriculum of the new Nottingham vet school was designed to reflect an important report into these current issues in veterinary education, the Pew national veterinary education program [4], which was concerned with veterinary education in the USA and Canada, and the foundation statements of EAEVE in 1990. These outlined a need for a change in how veterinary education is delivered with an emphasis on a deeper understanding of the scientific principles which underpin clinical training, rather than a passive recall of facts, an emphasis on problem solving and professional and personal skills training. A need for lifelong learning across the whole University sector was also highlighted by other influential reports such as the report of the National Committee of Inquiry into Higher Education (Dearing report) in 1997 [5] and the report of the National Advisory Group for Continuing Education and Lifelong Learning: Learning for the 21^st Century (Fryett report) in 1997 [6]. These reports also needed to be considered when developing the Nottingham course.

The impact of increased course content on lower order learning can lead to a reduction in both the retention of knowledge and the ability to apply this knowledge in a clinical setting, which is in conflict with the aim of generating lifelong learners in clinical science, since poor self-learning skills are promoted [3]. Students can, however, respond to the increased challenge of a course, with relevant resources in place, by recognising that they are required to work as more independent learners [7]. This type of educational philosophy requires that students learn how to learn and the role of a course organizer is to facilitate this transition to educational autonomy, by providing clear educational scaffolding [8]. The Pew and Dearing reports did not set out a prescriptive educational mechanism for the achievement of its aims. In an attempt to achieve these aims, the Nottingham veterinary course was developed along a radically different pathway to other UK veterinary degree courses by utilising vertical and horizontal integrations of subjects. In this model the course is divided along body systems rather than discipline-based modules and interwoven within these systems based modules are integrated modules which are subject based, for example immunology or microbiology may be taught within the larger systems-based Gastrointestinal system module (Table 1). The inclusion of integrated modules is designed to place the learning of these subjects in context, which may promote a deeper understanding of them. This format of teaching represents a major shift from the traditional teaching philosophy of established UK veterinary schools in which fundamental science delivery is built on dedicated modules with signposting lectures and practical classes.

The increasing workload and attempts to motivate and deepen students learning has led to pressure for a more integrative approach to teaching within medical sciences. Integration reduces the isolation of component subject areas by integrating them under applied and clinical themes [9-10]. In fact, some commentators have suggested that the level of course integration can be used to assess innovation in medical education [11]. This 'integration-ladder' ranges from the complete isolation of subjects at its lowest level up to the complete interdisciplinary integration of key concepts. This trend effectively links the integration ladder to a more traditional learning taxonomy such as that proposed by Bloom [12] or, more appropriately, the revised taxonomy proposed by Anderson and Krathwohl [13], since the latter includes evaluation and creativity as higher learning levels which require a metacognitive awareness of subjects to improve practice. Thus, the more we can integrate the deeper the understanding which can be achieved. However, although we have a rationale for wanting to change we need to know if this change in educational philosophy has a pedagogical basis, as John Dewey suggested there is danger in developing a new education simply because we wish to offer something different from the old system [14]. This may be the case as the integration of the course may marginalise specific areas and lead to a more superficial learning experience, discussed below. The aims of the study were to analyse student perception of integrated modules and assess and compare student performance in both body-systembased and integrated subjects when taught together. Thus, to ascertain whether the novel teaching programme at Nottingham undermines or enhances delivery of basic sciences in a clinical context.

Two student cohorts were chosen; these were cohort 1, which consisted of 101 students, which had begun the third year of the course and cohort 2, which consisted of 97 students which had completed year two of the course but had not yet been assessed in three year modules.

The performance of both cohorts were assessed against two systems based modules which contained integrated modules, these were the cardio-respiratory system (CRS) (Year 1) module

and gastrointestinal system (GIS) (year 2) module and in addition to these, the response of cohort 1 to short answer questions was also assessed in the principles of veterinary science (PVS) (year 3) module, which consists entirely of integrated modules and is designed to consolidate and build on previous knowledge of integrated modules prior to students entering the clinical curriculum (at the end of year 3) (Table 2).

To assess the effect of module integration on learning, we compared student responses to questions which assess different knowledge levels, as proposed by Anderson and Krathwohl [13]. These are: level 1. Remember (recall); level 2. Understand; level 3. Application; level 4. Analysis; level 5. Evaluate and level 6. Create. To study student performance at these cognitive levels we analysed their performance in three types of assessment format which broadly measure the different levels of knowledge in both systems-based and integrated subject-based questions. Therefore, analysed multiple choice questions (MCQs) which have been described as promoting and reflecting lower order learning [15], assertion-reason questions (ARQs), which have been shown to be a good predictor of student performance in essay work and reflect higher order learning [16], and finally we analysed student performance in essays.

Equal numbers of questions were analysed when comparing the student response to systems based and subject based (integrated) questions. Therefore, direct comparison could be made between the mean numbers of correct answers obtained from each individual response in a cohort divided by the number of systems-based or integrated questions. Thus, if a cohort consists of 100 students and all students answer 5 systems based questions correctly but only 50 answered 5 integrated questions correctly the mean number of correct response would be equal to 100 and 50 respectively. However, in some cases the comparison of responses to ARQs could not be made due to the large disparity between the numbers of systems-based ARQs and integrated ARQs. Data obtained for student performance in GIS essay questions was expressed as a percentage and was not subject to statistical analysis.

Minitab was used to analyse all quantitative data. Data sets which required comparison of 2 means were analysed by student t-test and data sets which compared more than two means were analysed by Analysis of Variance (ANOVA) with a one-way classification. Any significant difference determined by the F test was further analysed post hoc by Tukey's test (to determine where the significance occurred). All values obtained from statistical tests were compared with tabulated values at P = 0.05 (95 % confidence limit) to determine whether or not they were significant.

An Online Focus Group (OFG) was used to investigate the perception of integrated modules by undergraduates. OFGs have previously been shown to provide comparable information to conventional ('face-to-face') focus groups but with the added advantage of convenience, since there is no time constraint, and participants are more likely to honestly disclose their views [17- 20]. Twelve undergraduate students from year 2 and year 4 were randomly selected for inclusion with the rationale that second year undergraduates had experience of integrated modules from their first year modules and that fourth year undergraduates have experience of integrated modules from years one to three and year four, which includes a much stronger components of clinical academic learning and practical learning via Clinical Extra-Mural Studies (CEMS). Fourth year undergraduates may, therefore, have similar experience to second year undergraduates regarding the use of integrated modules in non-clinical years but have greater experience and possibly a broader opinion obtained from CEMS and the deeper application of integrated knowledge. Ten, second year students (including 2 males) and eight, fourth year students (including 1 male) responded to the questions. Two focussing on topics were included and both topics contained two linked questions. No prompts were used (other than linked questions) to minimize bias which may have occurred due to the over-direction of student opinion. The following questions for discussion were sent by email to all participants en-bloc to ensure that each student had the names and email addresses of other participants within the group:-

(1) Please express your opinion regarding the importance of module integration in undergraduate teaching and learning.
(A) Do you recognise an integrated module when it is taught within a larger system-based module?
(B) What is your view of module integration?
Please feel free to provide any information you wish and to discuss these points with other participants.

Ten, second year students and eight fourth year students responded to the questions.

Only two, second year undergraduate students said that they specifically recognised an integrated module when it was being delivered within a larger systems-based module, although all students gave an example of an integrated module when they expressed their opinion. The two who were conscious of an integrated module being delivered stated that:

(1) 'The integration within the modules is very good and the repetition helps with deeper understanding e.g. the immunology lectures at the end of last term helped a lot and gave an excellent chance to revise our understanding'.

(2) 'I think for many of the integrated modules fitting them into the long modules is a better option than having them as separate modules as they often involve a lot of memorising. For example, I think microbiology would be too overwhelming if it were all taught in one long module and it would be a lot harder to remember all of the information'.

All other students in this group said that, they did not specifically recognize an integrated module and regarded them as being integral to the bodily system being taught (the larger systems-based module). One student, who said that she did not recognize independent integrated modules within systemsbased modules, qualified her opinion as follows:

'I hadn't personally noticed integrated modules, but I personally find it quite difficult not to think in module mode.'

There were some very interesting differences in the opinions of fourth year undergraduates compared with second year undergraduates. All students in this group stated that they paid little attention to an integrated module other than in the context of the larger systems-based module but, as with second year students, all fourth year undergraduates gave an example of an integrated module when expressing an opinion. In this regard the opinion of fourth and second years was comparable, however fourth year undergraduate students appeared to be more aware of an integrated modules but chose to ignore them, as shown by two of the testaments below:

(1) 'I don't really notice much integration, but I'm not looking for it. I'm sure it's a good thing to do because it helps you in the real world and helps revision of other subjects.'

(2) 'If asked to find the integrated modules then yes I could probably point them out. However, I tend not to consider the modules as distinct sections of the course, rather just knowledge acquired during the course that needs to be applied.'

However, one student also expressed a preference for integration of some modules rather than the traditional way these are taught (as stand alone block modules) in other UK vet schools. This student believed that integration of some modules helped to contextualise them within body systems and stated that:

'I think we all accept that, everything is integrated so we tend not to appreciate it as much! I like the fact that parasitology and microbiology are introduced at the relevant points in the course rather than stand-alone modules because it highlights the relevance of them.'

To assess whether module integration affected the retention and comprehension of knowledge, we examined student performance in both CRS (year 1) and GIS (year 2) and compared these with the performance of students in integrated modules, delivered within the systems-based modules. Student performance was assessed by the numbers of correct responses to MCQ (knowledge retention; level 1) and ARQ (knowledge understanding and application; levels 2-3) and essays (which can actually assess all 6 levels of Anderson and Krathwohl's taxonomy, depending on the information they require).

For this group only MCQ data existed since only one ARQ from an integrated module was used during the examination and therefore, comparison could not be made between the responses of students to ARQs from the systems-based module and the integrated modules contained within this systems-based module. Means were calculated from 5-16 questions per integrated module (depending on the number of questions chosen from each module to be used in the examination). However, when integrated modules were combined the mean responses were calculated from 16 and 15 questions respectively. The data shows that there was consistency between the numbers of students who responded correctly to MCQs from the systems-based module and from all integrated modules. When this latter group was taken as a whole and the responses to MCQ questions was compared to responses to MCQs from the systems-based module, or when each integrated module was compared independently to each other and to the systems-based module, there were no significant difference between the groups (P > 0.05) (Figure1).

The data sets generated from the second student cohort contained fewer questions overall from integrated modules (5 MCQ and 5 ARQ) but the inclusion of some ARQ questions allowed some comparison between the systems based module and the integrated modules with respect to higher knowledge levels. In this data set all of the integrated questions actually came from one module (microbiology) (Figure 2).

Although the data collected showed a trend towards reduced correct responses in systems based ARQs compared to ARQs from the integrated module, and conversely increased number of correct responses in systems based MCQs compared to MCQs from the integrated module, there were no significant difference (P > 0.05) between the numbers of correct responses by students when comparing between ARQs or MCQs in the systems-based module or ARQs and MCQs in the integrated module, or between different modules (systems-based compared to integrated) (Figure 2).

The next part of the study was designed to compare the responses of students to short answer questions from systemsbased and integrated modules in the second year (gastrointestinal system) module but the two student cohorts analyzed were the same cohorts analyzed in the CRS. Data from 8 systems-based MCQs and 11 integrated MCQs (3 parasitology; 6 microbiology and 2 immunology) were analyzed. Only two integrated ARQs were included in the examination, therefore, comparison of the student response to ARQs from systems-based modules and integrated modules could not be performed. In this data set there were significantly fewer (P < 0.05) correct responses to Microbiology MCQs compared with systems-based MCQs, but there was no significant difference between the mean response to parasitology or immunology MCQs. However, in some cases the standard deviation of mean responses was very high and, as previously stated, some integrated modules were represented by only 2-3 MCQs. Nevertheless when combined, the response to integrated modules was not significantly different to those for the systems-based module and the difference was closer if

microbiology questions were removed. The response of cohort 1 to these microbiology MCQs was much lower than for other integrated modules (Figure 3).

Data from this cohort included 8 MCQs from the systemsbased module and 15 MCQs from integrated modules (10 from parasitology and 5 from microbiology) as well as 7 systemsbased ARQs and 5 ARQs from different integrated modules (split between parasitology, microbiology and immunology).

In accordance with data obtained in the CRS module, data obtained in the gastrointestinal system module also showed a trend towards a reduction in the numbers of correct responses in systems-based ARQs compared to ARQs from integrated module, and increased number of correct responses in systems-based MCQs compared to MCQs from the integrated modules (Figure 4). However, these trends were not significantly different (P > 0.05) when compared to the numbers of correct responses between ARQs or MCQs in the systems-based module or ARQs and MCQs in the integrated module, or either type of question between different modules (systems-based compared to integrated) (Figure 4).

The principles of veterinary science module combine integrated modules and is designed to pull together all integrated teaching delivery prior to the students beginning the clinical curriculum. In this study, three of these modules (microbiology, parasitology and immunology) were analyzed to determine whether or not students respond differently in a module which has no systems-based teaching. The data analyzed consisted of both MCQs (8 microbiology; 7 parasitology and 8 immunology) as well as ARQs (5 microbiology; 3 parasitology and 5 immunology). The data shows that, there were significantly lower numbers of correct responses (P < 0.05) to parasitology ARQs compared to all other questions. There was no significant difference between the responses of students to any of the other questions taken from different modules (Figure 5).

Knowledge-based essays simply measure the ability of students to recall facts and to a lesser extent understand and apply these facts (level 1 to 3 knowledge) and there are no specific instructions were given regarding content or format. In contrast, essays based on knowledge integration require students to recall knowledge but then to place this in context across the whole of the module and as such measures a deeper understanding of a subject via the ability of students to apply, analyse and evaluate knowledge and then to use these to create (reorganize and represent) knowledge (knowledge levels 3-6). Students are specifically instructed that knowledge integration

essays are designed to assess the ability of students to integrate material across the module and that students will receive marks for the selection and integration of appropriate material, as well as the structure and style of the essay.

Students were placed into cohort 1 or cohort 2 depending on the year in which they studied the GI modules.

Cohort 1 was asked to write essays on the following questions: 1. Knowledge-based question (Integrated module, Parasitology)
2. Knowledge-based question (Integrated module, Biochemistry)
3. Knowledge-based question (Integrated module, Microbiology)
4. Knowledge integration based question (Integrated module, microbiology)
5. Knowledge integration based question (system-based module)
6. Knowledge integration based question (system-based module)

Results from these assessments suggested that students in cohort 1 generally performed better when asked to integrate knowledge from sources across the entire module. Students performance in knowledge integration essays was similar to or above the overall mean performance and one of these questions was from an integrated module (microbiology) (Figure 6A). However, when this cohort were asked to answer knowledgebased essays only one of the questions scored comparatively to the overall mean and all three questions were taken from integrated modules and the equal lowest mean score attained (30%) in knowledge-based essays was also from microbiology (Figure 6A).

Cohort 2 was asked to write essays on the following questions: 1. Knowledge-based question (Non-integrated module)
2. Knowledge-based question (Integrated module, Microbiology)
3. Knowledge-base
d question (Integrated module, Biochemistry) 4. Knowledge integration based question (Non-integrated module)
5. Knowledge integration based question (Non-integrated module)
6. Knowledge integration based question (Integrated module, Parasitology)

Results obtained from these assessments indicated that students in cohort 2 performed similarly in essays which were knowledge-based or those which required knowledge integration (Figure 6B). In this analysis mean values of all questions would have been even closer to the overall mean if one question

(question 5) was removed from the data set, since students scored very highly in this questions and this elevated the overall mean.

Another interesting observation when comparing cohort 1 and 2 data was that when one, almost identical, question which was taken from an integrated module (parasitology) required a knowledge-based answer (cohort 1, question 1) or knowledge integration (cohort 2, question 6) the mean result was almost identical with mean values of 53% and 50% respectively (Figure 6A,6B). Another question which was almost identical (cohort 1 question 2 and cohort 2 questions3) in both years was knowledgebased but results regarding student performance showed that the performance of cohort 2 was almost twice as high as those for cohort 1 when answering the question.

For the OFG, the recruitment target was set at eight students in accordance with studies by Chapple and Murphy, [21] who reported that this number was enough to provide meaningful data. However, Krueger and Casey, [22] reported that a 20% over-recruitment target should be aimed for to take into account the average drop out, or initial non-compliance, rates from these studies. Therefore, we randomly selected twelve students (25% more than required) from second and fourth year undergraduate lists. Ten, second year students and eight, fourth year students responded to the questions. This in itself was an interesting result since it represented a major increase in student response compared with our previous study which had used a conventional (rather than OFG) technique [23]. The views of students regarding module integration were surprisingly consistent in the different year groups with all students perceiving integrated modules as being a component part of the more dominant systems-based module rather than a 'stand alone' module within a systemsbased module. However, there tended to be a more conscious recognition of module integration by fourth year undergraduates compared to second year students. The view of our cohorts mirrors that reported in a previous study with a biomedical student who appreciated the applied nature of the course and the benefits of contextualization within the course [24]. This contextualization has a pedagogical value similar to that described for medical courses by Regehr and Norman, [25]. Our study shows that, students in both non-clinical and clinical years have an appreciation of whole body systems and this may have great importance once these students qualify. Veterinarians focus on the affected body systems and then consider which agents and factors lead to resultant pathology. Contextualization of the relevant knowledge to specific body systems may, therefore, aid the practitioners recall as it is already linked to cases rather than having to pick specific knowledge from stand alone block modules. Thus, integration may facilitate student development towards a more holistic approach to clinical cases which is required in practice. As such, the cognitive depth may also be increased since a broad knowledge of a body system (including the relevant micro-organisms that may infect it and the likely immunology and pathology which may be highly specific to that body system) may be in easier 'cognitive reach', similar to the coherence principle discussed by Mayer, et al. [26].

One question we wanted to answer here was whether the depth of knowledge was affected by the integration of subjects compared to the depth of knowledge attained within the main module. We carried out a quantitative analysis of student performance in different cognitive level assessments. MCQs are generally favoured greater than open-ended questions such as essays [27] but are usually used to assess knowledge retention [28] which is the lowest level of knowledge (level 1) according to Anderson and Krathwohl, [13], while ARQs and essays may assess higher levels of knowledge [16,29,30]. In some cases, the lower proportion of ARQ questions included in final examinations made a comparison of student performance, with ARQs and other formats, impossible to measure. Overall our study shows that students did not perform differently when answering different question types (which assessed different knowledge levels) when comparing questions requiring knowledge of integrated modules or systems-based modules. However, we have previously reported that students are less successful when answering ARQs compared to other short answer question formats but this is a general phenomenon and is not associated with integrated versus non-integrated content [31]. Thus, students do not learn systems-based areas of a module to the detriment of integrated modules contained within these systems-based modules, which has been a concern of integrated module convenors. This may be explained by the student perception of integrated modules (as discussed earlier) which indicated a focus towards systemsbased modules as a whole unit, and a perception that any teaching within these modules (independent scientific disciplines which form integrated modules) were not considered to be separate 'tag-on' subjects. This may benefit students once they graduate because it suggests that they are learning integrated subjects in the context of the body systems to which they apply. Integration also allows a re-visitation (and re-evaluation) of knowledge since certain facets of a particular integrated discipline may appear in more than one body system. For example, specific immune cell types are activated in response to specific pathogens in all body systems and this allows the teaching of the biology of these immune cells to be repeated and expanded when appropriate, which probably has a positive effect on knowledge retention (by a certain degree of repetition) while building on more basic levels of knowledge as the integrated module develops. This spiral curriculum, previously proposed by Bruner, [31] is further reinforced by some of the same first and second year modules being taught, at a higher cognitive level, in the fourth year, and is therefore also in accordance with the concept of cognitive hierarchies [12,13]. Therefore, module integration may promote a smoother cognitive connection between different disciplines, rather than an assembly of disconnected disciplines into a rational order, prior to solving a scientific/clinical problem. However, this cannot be confidently proposed unless we compare the response of students from non-integrated (traditional) veterinary courses to the same questions, particularly those which specifically require integrated knowledge (such as year two GIS essays). Nevertheless our study at least suggests that students almost subconsciously learn disciplines which are integrated within larger modules.

In conclusion, our study provides some pedagogical evidence to suggest that module integration may have a beneficial effect on undergraduate education in veterinary science and medicine.

All datasets were analyzed with the approval of both Nottingham school of Veterinary Medicine and the University of Nottingham.

1. Davis M.H. (2003). Outcome-based education. Journal of Veterinary Medical Education.  30: 258-263
2. Bushby PA. Tackling the knowledge explosion without overloading the student. Aust Vet J. 1994;71(11):372–374.
3. Blumberg P. Why self-directed learning is not learned and practiced in veterinary education. J Vet Med. Edu. 2005;32(3):290-295.
4. Pritchard W.R. Future Directions for Veterinary Medicine. Durham, NC: Pew National Veterinary Education Program, Institute of Policy Science and Public Affairs, Duke University; 1988.
5. The Report of the National Committee of Inquiry into Higher Education, Higher Education in the Learning Society, 1997.
6. Report of the National Advisory Group for Continuing Education and Lifelong Learning: Learning for the 21st Centuary, 1997.
7. Crook C, Gross H, Dymott R. Assessment relationships in higher education: the tension of process and practice. Br. Educ. Res. J. 2006;32:95-114.
8. Sadler DR. Perils in the meticulous specification of goals and assessment criteria. Asses. Educ. 2007;14:387-392.
9. Turnwald GH, Banks WJ. Problems in veterinary medical education: A case for increased integration of basic science and clinical curricula. J Vet Med Educ. 1991;18:25-30.
10. Prideaux D. Curriculum development in medical education: from acronyms to dynamism.Teaching and Higher Education. 2007; 23: 294-302.
11. Harden RM, Sowden S, Dunn WR. Educational strategies in curriculum development: the SPICES model. Medical Education. 1984;18:284-297.
12. Bloom BS. Hastings JT, Madaus GF. Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook 1, Cognitive Domain. New York: McKay; 1956.
13. Anderson LW, Krathwohl DR (eds.) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman; 2001.
14. Dewey J. Experience and education. MacMillan: New York;1983. p. 6.
15. Leamnson R. Thinking about teaching and learning. Sterling, VA, Stylus Publishing. 1999; p.111.
16. Connelly LB. Assertion-reason assessment in formative and summative tests: results from two graduate case studies, In: R. Ottewill, E. Borreddon, L. Falque, B. Macfarlane and A. Wall (eds.) Educational innovation in economics and business VIII: pedagogy, technology and innovation (Dodrect, Kluwer Acadmeic Publishers); 2004.p.359-378.
17. Montoya-Weiss MM, Massey AP, Clapper DL. On-line focus groups: conceptual issues and a research tool.  European Journal of Marketing. 1998;32:713-723.
18. Mann C, Stewart F. Internet communication and qualitative research: A handbook for researching online. London: Sage Publications; 2000.
19. Ahern NR. Using the Internet to conduct research. Nurse Research. 2005;13:55-70.
20. Reid DJ, Reid FJM. Online focus groups: An in-depth comparison of computer-mediated and conventional focus group discussions.  International Journal of Market Research. 2005;47:131-162.
21. Chapple, M, Murphy R. The Nominal Group Technique: Extending the Evaluation of Students Teaching and Learning Experience. Assessment & Evaluation in Higher Education. 1996;21(2):147-158.
22. Krueger RA, Casey MA. Focus groups: a practical guide for applied research (3rd ed.). Thousand Oaks, California: Sage Publications; 2000.
23. Roshier M,  Foster N, Jones MA. Student Perception of Teaching Video Resources. BMC. Med Edu. 2011;11: 1.
24. Barragan EI, Alejandra M, Guillermo DH. Horizontal integration of teaching within a biomedical department. Medical Education. 2005;39(11):1143-1172.
25. Regehr G, Norman GR. Issues in cognitive psychology; implications for professional education. Academic Medicine. 1996;71(9):988-1001.
26. Mayer RE. Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction". Am Psychol. 2004;59(1):14-9.
27. Zeidner M. Essay versus Multiple-Choice Type Classroom Exams: The Student's Perspective. Journal of Educational Research. 1987;80(6):352-358.
28. McGuire C. Written methods for assessing clinical competence. In: Hart I. and Harden R. (Eds). Further Developments in Assessing Clinical Competence. Can-Heal Publications;1987.
29. Centre for Computer Assisted Assessment (CAA). Assessment of higher order skills. 2002 http://www.caacentre.ac.uk/resources/faqs/higher.shtml. Accessed 18/02/2011.
30. Dunleavy P. Studying for a degree in the humanities and social sciences. London: Macmillan Education; 1986.
31. Foster N. Analysis of short answer question styles versus gender in preclinical veterinary education.  J Vet Med. Edu. In press. 2011;38(1): 67-73.
32. Bruner J. The Process of Education. Cambridge, Mass.: Harvard University Press;  1960.