We describe here a scheme for testing and assisting students with the mathematical skills we think are essential for their courses. These are the skills that lecturers assume students have mastered before a course begins. We discuss the aims and philosophy of the scheme and describe the logistics of running such a scheme. We also look at the performance of the students and discuss the skills that students mastered or have difficulty mastering. We ask for feedback on the skills you think are essential to a mathematics degree. For those interested in implementing similar schemes, our WWW site, http://www.ma.adfa.oz.au/~sib/Level2ems.html , has chapters of the notes, a database of questions, and tests which we have made available (in LaTeX and postscript).
\frac{1}{y} = \frac{1}{x} + \frac{1}{c} \Rightarrow y = x + c
Have you ever marked an exam and had a student write something like this? We have! And not just once. And not just first year students. This level of error occurs frequently in the work of our students and most heart-breakingly in exams and tests. Whether the source of error is true misconception or simple carelessness, students are still held back from progressing to more advanced work.
If your department is anything like ours, endless hours in the tea room have been spent discussing the perceived decline in our students' basic skills and the mathematical preparedness of our first year students. Evidence of such concern is found in Australia [9] but has been an international concern for some time [2,3,5,7,8] particularly in the United Kingdom with the publication of a major study commissioned by the London Mathematical Society [6].
In 1995, in an attempt to tackle the problem, the Essential Mathematics Scheme (EMS) was initiated. Very briefly, the EMS is a set of notes, a series of tests and the provision of help. The notes clearly list the techniques and skills we assume students have mastered. Every student enrolled in a mathematics course (in all years) is given a test at the beginning of each semester. The questions are multiple-choice and the pass mark is 80~%. If a student fails the initial test they must re-sit a similar test and keep re-sitting (up to 3 more times) until they pass.
The purpose of this article is to relate our experiences with the EMS, provide some insight into the problems students have and give you ideas for how to help your own students who may have similar difficulties. At this stage we are not attempting to give a rigorous statistical proof that students' abilities are declining [5] or that the education system is failing.
In Section 2 of this article we discuss the aims of the scheme, the EMS notes, the types of questions we ask in our tests and also similar testing schemes used in the United States of America. We then evaluate the performance of our own first year students and focus on the areas of mathematics where these students are particularly weak. In section 4 we give details of how we run the EMS and give guidelines for those wishing to set up their own scheme. This is followed by a general discussion and conclusion. An appendix with different categories of material and sample questions is also included.
The primary aim of the Essential Mathematics Scheme (EMS) is to help students. Not only to help them get better marks in their courses but also to remedy weaknesses in their mathematical background which aren't addressed in regular courses. There are also many secondary aims. Generally the scheme aims to:
The EMS notes clearly describe the material that is be tested, fully worked example tests and sample tests for students to test themselves. The notes do not contain any explanations, wordy descriptions or multitudes of examples. In most chapters we describe the facts we want them to remember and the techniques we want them to know. One example is given per technique. The notes are not meant to teach. They are simply reminders to students of what they may have forgotten and what we want them to remember. In this way they are ` between' a text book and a mathematical handbook. That is, a textbook gives the full explanation to a student but we do not expect a student to remember everything they learn in the text. A mathematical handbook is a list of things most of which we do not expect a student to remember. The EMS notes are a list of things we do want them remember. The EMS notes are, to our knowledge, the only book which fills this role. An example of the notes we give students can be viewed on the WWW site: http://www.ma.adfa.oz.au/~sib/Level2ems.html . We are currently preparing the notes for publication as a small book.
2.2 What is to be considered Essential?
We consider material essential to a course if it is: assumed by lecturers; not taught in lectures (although taught in previous courses); not rewarded in exams; sometimes done incorrectly causing loss of marks. However, what material should be included as ` essential' is a personal judgement of the lecturer concerned. We made considerable efforts to talk to the individual lecturers to find out what they assumed for their courses. Some inevitable discrepancies arose, such as when the second and third year lecturers insisted that Taylor series should be included in the EMS until we found out that no-one taught the students Taylor series! (They are now taught in Mathematics 1). This sort of problem arose repeatedly as what one lecturer thinks is essential in one course, is barely considered important by a lecturer in the preceding course. The EMS has thus changed, and will continue to change, each year as different skills are emphasised. Most of our courses revolve around applied mathematics (continuum mechanics, waves, differential equations etc) and statistics - with many of our students being engineering and science students. The EMS reflects this. As such we have not included `` pure mathematics" topics.
The first year skills we felt were essential revolved around algebraic manipulation (simplification, quadratics, exponentials and logarithms, inequalities) and basic calculus (differentiation, product rule, chain rule, basic integration). In second semester we also tested the first year students basic vectors (dot product and length). For second year students we felt the students should be capable of slightly harder algebraic manipulation and differentiation questions and basic matrix questions (determinant, inverse of 2\times 2 matrices, eigenvalues and eigenvectors), simple separable first order differential equations and simple second order differential equations (with constant coefficients). Appendix A has a list of all the types of questions we ask and the full set of questions is available on our WWW site.
A few universities in the USA have similar schemes called Gateway Testing. For example, The United States Military Academy, West Point [1] and The University of Michigan [7]. The gateway tests (also called barrier tests) are usually focussed on first year calculus and are proficiency tests that must be passed as part of the course requirement. They are designed to complement the ` Calculus Reform' method of teaching which does not emphasise mastering of essential skills. These schemes run with much the same philosophy and aims as the EMS but differ slightly in how they operate. For instance the University of Michigan [7] offer 5 tests (each of five questions) which students can sit and re-sit. For each test failed their final mathematics mark is reduced by one grade, for example they drop from a B^+ to a B. It is interesting to note that the University of Michigan encountered similar difficulties as we did in formulating their scheme and came up with similar solutions (such as restricting the re-sitting time to the first few weeks of semester).
The material that students found difficult often surprised us - particularly the first year we ran the scheme when we were forced to rapidly drop the pass mark to 70% to avoid failing the entire class! In this section we look at the results of the tests and the material that students typically have problems with.
In the first week of teaching, every first year student sat one of a set of 5 Essential Skills Tests. These tests are written using a database which contains questions divided into categories that test the same skills or knowledge. Thus question 3 of test 2 is designed to test the same skill or knowledge as question 3 of test 4. The tests consist of 15 questions which are divided into 3 equal sections. The first five questions (section A) deal with basic algebra such as factorisation, the quadratic formula, and simplification of logarithms, exponentials and algebraic expressions. The next set of five (section B) ask basic differentiation and integration questions. The last five questions (section C) ask various mixed skill questions such as inequalities, expansions and basic trigonometric formulae.
Figure 1 shows the percentage of students correctly answering each of the fifteen categories. For each category we have averaged performance over the five tests. We have also included a horizontal line indicating the pass mark. As for previous years the results for the EMS have been somewhat disappointing; students have real problems with the material that we expect them to have mastered and retained. There are few categories of questions that sit above the pass mark and a number of categories that lie well below. To aid understanding of Figure 1 a complete list of categories (and questions that typify them) can be found in Appendix A.
\midspace{11cm}
It is interesting to consider both the categories of questions that the students found most difficult and individual questions which stood out from within their category as being difficult. Two categories of questions (questions 3 and 4) and three individual questions (questions 2 and 11 of test 3, and question 5 of test 1) drew a considerable proportion of incorrect responses. The two categories of questions that students found difficult were (with the average percentage of students answering correctly, indicated):
$\bullet$ Algebraic Simplification. For example:
`` Rearrange the following equation to find x: \sqrt{\frac{1}{x} - \frac{1}{y}} = 3."\quad (53%)
$\bullet$ Logarithmic and Exponential Expressions. For example:
`` If \ln x = 7 and \ln y = 2 evaluate: \ln \left( \frac{x^2}{y} \right)." \quad (53%)
The three individual questions that students had problems doing (with the percentage answering correctly indicated) were;
\bullet Question 2, test 3. Solve for d if -2d^2 + 7 = 4d. \quad (38%)
\bullet Question 5, test 1. Simplify \left(\frac{x-1}{x+1}\right)\left(\frac{x}{x-1} \right) - 1. \quad (50%)
\bullet Question 11, test 3. Solve the following inequality for t: |3t-1| \leq 7. \quad (43%)
The fact that less than half of our students can correctly answer questions such as those listed above is alarming. These poor results may be due to a number of outside factors such as: the three month break between leaving school and beginning their course at ADFA; the stress of entering university; or the exhaustion associated with the first few weeks of military life (the students are put through a rigorous and tiring training program at ADFA prior to the beginning of the academic year). However, these results are poor relative to performance in other areas tested by EMS (most notably Calculus). Hence it is likely that these results reflect real deficiencies. The results of re-testing also lead us to believe this conclusion.
The questions and categories listed above have in common that they require fundamental simplification skills. One of the outstanding categories (and one of the questions) specifically tests skills in algebraic simplification and the obvious conclusion is that these students find algebraic simplification difficult.
In this section we will discuss in a little more detail the logistics of the scheme. We will also attempt to impartially discuss its problems and successes.
4.1 Logistics of Running the EMS
The students are given the EMS notes a few weeks before semester. These contain practice tests for the students to attempt. The EMS test is given in the first tutorial of each course and takes about - 40 minutes for them to complete. The test comprises three sections each with five questions. The test is multiple choice although we still make sure the appropriate working is correct. Generally about 60% of students have to re-sit one or more sections. This they do in their own time in a Mathematics Study Centre set up in our School. This Centre is staffed for two hours a day, every day of the week, and was initiated by Dr Peter McIntyre to enable students to have a place to work, with a tutor ready to answer any questions.
We produce up to 5 different tests for each year from a central database of questions (using LaTeX). The tests are arranged so that no student sits the same test twice. The test results are recorded on a central database and student results for individual sections questions collated for survey purposes. We consistently remind the students to re-sit test as soon as possible.
4.2 Benefits of the EMS
Many of the benefits of the scheme are given as part of the aim. It helps us identify weak students, lets them realise their own weaknesses, helps us appreciate areas where our students (as a whole class) are weak and informs the student of the techniques we want them to know.
One of the primary benefits, however, is that it forces weak students to come and talk to us. After each test we get a few minutes to discuss their difficulties and help them with fundamental problems they may otherwise never get help with. For example, we know of several second year students who for over a year had been using a wrong method to take the cross product of two vectors and were not aware of their error until tested in the EMS. Similarly we have spent many hours helping students with the fundamentals of chain rule differentiation or even basic fractions. While anecdotal evidence is not proof, we are confident that the time we spend with these students on these skills has been very important to them and that without the EMS we would not have been aware of some students difficulties or been in a position to help them.
The EMS also has another major benefit. We know that by the end of the testing period (6 weeks) every student, has at least once been able to do the skills we ask of them. This may not be entirely true since the pass mark is only 80 %. However, in regular mathematics courses students may attempt (and often succeed) to pass exams by only learning 50 % of the material - they may choose to ignore particular skills and still easily pass the course. This is much more difficult to do with the EMS scheme and gives lecturers confidence in assuming students have the skills they require.
We have also found the EMS notes to be very successful beyond the confines of the Scheme. Students often refer to them to remind themselves of a particular formula or technique. We have also had other schools within the University using the notes as a guide to what they can assume students know or do not know.
4.3 Negatives of the EMS
There are three main negative aspects associated with the EMS:
(i) Consumption of student time: The EMS does take time for some students since they have to do a bit of revision and spend time re-sitting the test. The students often see the EMS as an imposition, especially if it is not worth any marks. They think that the time spent on the EMS takes time away from their regular study. (As students are often motivated solely by marks they do not realise the importance of these essential skills within their usual mathematics course). While the EMS does take some time we believe this revision time is important. We must note that for the students who pass the initial test, in the first tutorial, the EMS takes no additional time away from their usual study. In a survey of students we found the majority indicated they spent only one hour over the entire semester on the EMS. No students indicated they spent more than 4 hours on the EMS.
(ii) Consumption of staff time: The EMS takes considerable time to run and set up. Each year, organising the tests, printing and associated administration takes approximately 1 week. During each of the first six weeks of semester there is approximately 1-2 hours spent in administration plus 3 - hours in personal student consulting (helping students with problems and dealing with students who cannot make it to the two hour Maths Study Centre). There must also be staff available to help the students sit the test. During the first year of running the EMS, the time we spent was considerably more than this. The University of Michigan [7] estimate they spend `42 grader/tutor- hours for each gateway test given a 100-student course, assuming that the students need an average of 4.2 tries to pass and that the grader/tutor requires about 6 minutes to grade and discuss each attempt'. We believe the workload of an EMS type scheme is approximately the teaching load of a level A (associate) lecturer for the six week duration of the scheme plus a two week preparatory period before semester.
(iii) Handling those who fail: Appropriate penalties for failing to complete the EMS is a very contentious issue among our staff. If a student consistently fails the EMS tests then what should be done? Some staff believe the student should then fail the course which we believe is too harsh and would cause the students too much stress. Other staff believe that there should be no penalty, however, we have found that, due to student apathy, they then do not bother to participate in the scheme. As a middle approach the University of Michigan [7] reduce a student's mark by a few percent for every test not passed. At ADFA we do not punish any students who make a consistent effort to pass (those that try to pass three times) but any student who does make the effort has to do an intensive revision assignment and tutorial. Because of the military nature of ADFA we are able to contact the student's supervising officer to put pressure on them and hence we usually get 99 % of students completing the scheme successfully.
We have presented here the philosophy and aims of the Essential Mathematics Scheme. In the last section we gave a more detailed account of the logistics of running a scheme such as the EMS for those wishing to set up similar schemes.
The results of the scheme have outlined skills with which students have particular difficulty. While our results are not conclusive we found that there is a surprisingly large percentage of students who are unable to perform even the most fundamental of mathematical skills. We believe that through the EMS we were able to identify these students and give them the help they needed.
The issue of preparedness of students is a complex one at the Defence Force Academy as students are enrolled from different states and so arrive having worked through different curricula. Work is in progress which attempts to analyse the difference this may make to first year performance. This consideration also highlights the importance of having something like the EMS notes.
We would value your opinion of the EMS and what skills you think are essential in a mathematics degree. Please email us on s-barry@adfa.oz.au or s-davis@adfa.oz.au with your comments or if you want more information. Our WWW site:
contains copies of the notes, sample tests, the database of question and more information about the EMS.
The WWW site, http://www.ma.adfa.oz.au/~sib/Level2ems.html , contains a copy of the EMS notes and some sample tests which show the material we are trying to test.
The EMS is different for each year and for each semester of each year. Our courses are almost entirely applied mathematical. This is reflected in the EMS which restricts itself to those skills we believe necessary for applied mathematics, science and engineering students. The list of questions below constitutes a typical first year, first semester EMS test.
Section A: Algebra
Section B: Calculus
Section C : Mixed Skills
The list of questions below constitutes a typical 2nd-year, 1st-semester EMS test.
Section A : Algebra
Section B : Calculus
Section C : Mixed Skills
School of Mathematics and Statistics, University College,
University of
New South Wales,ADFA
Canberra ACT 2600, Australia.