The Presbyterian Church (U.S.A.) will give its next Bible Content Examination to seminary students on February 5, 2016 – and, despite low scores on the exam recently given, the plan is for the format to be much the same.
Only 28 percent of those who took the exam given Sept. 4 (or 36 of 127 students) earned a score of at least 70 percent – high enough to pass the test. In comparison, the last 12 times the exam was given online, the average of those who passed was 81.7 percent. The average score this time was 63.5 out of 100 points – roughly 10 to 15 percent lower than typical.
That result ignited a swirl of social media debate – about whether changes to the test were unfair; whether Presbyterian seminarians don’t know enough about the Bible; whether the test assesses what pastors really need to know in ministry or amounts to (as some have put it) “Bible trivia.” The Bible Content Examination (BCE) typically is given to students early in their seminary careers – so it’s considered more of a measure of what they come in knowing or study for early on, rather than what they’re taught in Bible or theology classes in seminary.
What happened – why were the scores so low in September? The biggest change, according to Tim Cargal, the PC(USA)’s assistant stated clerk for preparation for ministry: This time, no questions were used that had been on previous tests and were also publicly available. In other words, no luck boosting one’s score by memorizing old questions from previous tests.
The BCE test has been given online (rather than in a paper form) since 2009 – and since then, questions from the old paper versions of the test have gradually been filtered out. The test administered Sept. 4 did include a majority of questions that had been used on previous exams – but marks the first time that all those questions came from online versions of the test.
The change was made last March, Cargal said, because members of the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) kept hearing anecdotally that students “knew a sizeable portion of the questions were going to be questions that had been used in the past, so they were studying the past exams.”
In a statement sent Sept. 8 to Presbyterian seminaries and to presbytery Committees on Preparation for Ministry, Steve Ranney, moderator of the PCC and pastor of Tustin Presbyterian Church in Tustin, California, said the committee “remains committed to the previously announced plan to no longer use questions that are publicly available in order to encourage studying the Bible rather than past tests. We believe that as inquirers and candidates adjust their preparation accordingly,” the scores will rise.
The scores and the pass rate may improve. “But even more importantly,” Ranney continued, “our candidates for the ministry of Word and Sacrament will be more deeply grounded in the Scriptures that are foundational to our faith and will be able to use that knowledge to guide the churches to which they are called to grow in the grace of our Lord (1 Peter 1).”
Here are more of the details.
What’s on the test?
This 100-point exam included:
- Three questions in new formats, each with five items (for a total of 15 points). These questions appeared in either:
- A matching format, meaning a student would match each item from the left column with one item from the right column, such as matching a person from the Bible with the book in the Bible in which that person appears; or
- An ordering format, meaning the student would place the five items in the chronological order in which they appear in the Bible or in a particular book of the Bible.
- Two new multiple-choice questions (for a total of 2 points).
- 83 multiple-choice questions (for 83 points) that were repeated from previous online exams – but were not publicly available. Consequently the students couldn’t use those particular questions to study for the test.
What the test is not
In an interview, Cargal emphasized that the BCE isn’t considered a standardized test, like the ACT or SAT, but that the PCC attempts to make sure that each exam includes a mix of easier and more difficult items. The General Assembly has instructed that the BCE is to test knowledge of “stories, themes and key passages” of the Bible, Cargal said.
Test questions are repeated, he said, because that is a tool the PCC uses to judge the relative difficulty of particular questions (based on how well students did on those questions previously). For the February 2016 test, Cargal said, the PCC will assess whether it has the right mix of more difficult and easier questions on the test – but plans to continue using only questions that have not appeared in printed versions of the test.
Studying from old tests
When the tests were given in paper form, students were given their scanned answer sheets back – along with a copy of the test so they could see what items they had missed. Over time, seminaries began keeping files of the exams, Cargal said, and some tests were posted electronically, so students studying for the exam could access them.
One way to tell that sharing of old tests was affecting the scores was to measure how well the students did on a particular question the first time it was given, compared to when it was repeated subsequently. Consistently, Cargal said, scores on individual questions from printed tests rose by 5 to 10 percent each time when they were repeated. “Did people just get smarter?” he asked. “No, they go back and they’re studying the old tests.”
In online comments, however, considerable disagreement arose about how well this particular test was designed; whether students taking the test knew of the changes; and whether the lower scores reflect a lack of familiarity with the Bible – or something else. Some called on the PCC to privately release the exam – for example, to presbytery Committees on Preparation for Ministry – so more Presbyterians could judge whether the questions and the format were well-designed.
One student who said she passed the September exam and who described it as “very difficult” wrote on Facebook that “I don’t think my colleagues failed because they were all unprepared. Dropping from a 81 percent passing rate to a 29 percent passing rate indicates something besides test takers not knowing their stuff.”
In a comment to a blog post from Cargal, Darren Sumner wrote that he also passed the September exam but found it “surprisingly difficult,” even though “I consider my biblical literacy high” – Sumner said he holds four degrees in religion and theology, including a doctorate. Sumner wrote that “many questions are incredibly precise in their demands” and “in many cases the questions require having bits of more obscure minutia close to mind, and those aren’t questions that even a well-informed test-taker will get right without guessing.”
Cynthia Jarvis, who has served on the Committee on Preparation for Ministry in the Presbytery of Philadelphia, wrote in a comment to Cargal’s blog post that “I personally know four of the students who failed the exam and all four are serious students of the Bible. What are we testing?”
How to study
Cargal recognizes that some students today may come to seminary with less familiarity with the Bible than students typically might have had in the past – national surveys show a declining level of biblical literacy. His suggestions for studying for the BCE: develop a regular practice of reading the Bible; read the introductions from each chapter of a good study Bible, which often identify the key themes, and look up significant verses identified in those introductions. The PCC provides materials to familiarize students with the exam, and makes a practice test available about a month before the exam is given. In online comments, some students said they used children’s Bibles as a way of identifying themes, or took Bible survey courses to get ready.
How to use the results
Students who take the test no longer are given a copy of the exam afterwards, but are provided information on what questions they got right and what they missed (for example, they might be told they missed a question on a particular passage). The hope, Cargal said, is that students can use that information to identify strengths and weaknesses – strong on the Pentateuch, weak on the prophets, for example. “The value is the pattern of what the person misses,” he said – it may reveal the need to take a class focusing on a particular section of the Bible.
Bible trivia
The test is intended to measure key themes, stories and passages, Cargal said – it assesses whether “you can draw on the scriptural tradition as a tool of ministry.” The example he gave: a minister trying to answer the question someone poses over coffee of “what does the Bible say about concern for the poor?”
For those who say the BCE measures “Bible trivia,” Cargal responded that one of the questions the majority of students missed on the September exam had to do with the specific wording of one of the Ten Commandments. “Would we really say,” Cargal queried, “that was unimportant or an unfair question to ask, or that it was in some sense trivial?”
Click here to read the letter sent to seminaries and presbyteries from Steve Ranney.
