|
New York State Testing Program (NYSTP)
Frequently Asked Questions
About the Assessments
Q1.
Are 3-8 tests standardized?
A1. Yes. A standardized test is defined as a test that uses uniform
procedures for administration and scoring in order to assure that the
results from different people are comparable. The NYSTP requires
standardized administration. Tests must be administered exactly the same
way each time. Everyone must be given the exact same instructions in
sufficient detail that no differences in administration would take place
between settings or people who administer the test.
Q2. Are 3-8 tests norm-referenced or criterion-referenced
tests?
A2. They are criterion-referenced tests because they measure how
well students are meeting the learning standards in English Language
Arts and mathematics. In contrast, a norm-referenced test is made to
compare students to each other rather than determining whether or not a
student meets a certain criteria.
Q3.
Is there such thing as a bad test question? Exactly how is a test
question developed?
A3. A
hard test question is not necessarily a bad test question. NYSTP is
designed to determine whether students are meeting the learning
standards. Hard questions are those where only a small percentage of
students were able to answer correctly. The question may be intended to
have this characteristic in order to detect different student
achievement levels. Test questions, commonly referred to as items, are
developed through a process that involves many steps guided by industry
standards in assessment and measurement. Teachers are included in
committees that develop NYS tests to ensure that test questions are
aligned with the State’s Learning Standards and to help specify
performance indicators, question formats, and appropriate content for
the grade levels in which they are experts. This test specification
process leads to the writing of test questions following important
guidelines set by specialists. A rigorous review and editing of written
questions takes place many times during the process culminating in
statistical analysis of how the test questions perform on field tests.
For more information see
Guide to the Grades
3-8 Testing Program in English Language Arts and Mathematics
http://www.emsc.nysed.gov/3-8/intro07.pdf
Guidelines for Item
Writers:
http://emsc33.nysed.gov/osa/assesspubs/pubsarch/etsguidelines
Standard Setting and
Equating on the New Generation of New York State Assessments
http://www.emsc.nysed.gov/osa/assesspubs/pubsarch/standard%20settingand%20equatingon%20thenew%20generation.pdf
Educational Assessment: Four Principles to Consider
http://www.ctb.com/articles/article_information.jsp?CONTENT%3C%3Ecnt_id=10134198673246869&FOLDER%3C%3Efolder_id=1408474395243877&ASSORTMENT%3C%3East_id=1408474395213825
Q4.
Can results of previous ELA and mathematics testing for grades 4 and 8
be compared to results of the new NYSTP for grades 3-8?
A4. A representative of NYSED refers to the “old” and “new”
tests as cousins rather than brothers. The tests do not lend themselves
to absolute comparison because the “old” grade 4 tests included K-4 or
4-8 content and skills whereas the new 3-8 tests consist primarily of
specific material with a narrower range of content on each test. The
range of content covers what students learned at the end of the previous
grade level and at the beginning and middle of the current grade level
of the student. Of course, longitudinal and cross-sectional data will
become increasingly available in the coming years that will allow for
year to year comparison.
Interpretation of Results
Q5. What does it mean for an individual student to achieve a
Standard Performance Index (SPI) within the Target Range?
A5. It means that a student demonstrates the expected level of
understanding in the specific ELA Learning Standard or mathematics
Strand. An SPI is a derived score from 0 to 100 that estimates the
number of questions a student would have answered correctly if there
were 100 questions per Learning Standard or Strand. However, the way
tests are currently designed there are as few as four questions
developed to generate an SPI. The Target Range for SPI varies across the
Standards and Strands because the number and difficulty of the test
questions vary across these Standards and Strands.
Consequently, the SPI must be interpreted in context rather than as a
stand alone number. See: Interpreting Student Scores
http://www.emsc.nysed.gov/irts/nystart/2006/InterpretingStudentScores_files/frame.htm
Q6. Is it appropriate to use the scores of NYSTP as part of the
student’s classroom grade?
A6. No. Local districts may elect to display results on student
report cards. However, the scores from NYSTP are derived scores
and should not be calculated as percent correct or averaged with
existing raw score calculations. This is particularly confusing with
secondary level Regents exams, and elementary and intermediate social
studies and science exams, which use a scale score from 0 to 100. See:
How the Regents Exams are Scored
http://emsc32.nysed.gov/osa/concht/scoring-regents.htm
Q7.
The Northeastern Regional Information Center supplies my district with
reports of NYSTP results. At one time we received a report that detailed
which answers individual students selected on each test question. This
report was not distributed to us during the last reporting cycle. Why?
A7. Analysis of distractors, which are the
incorrect response choices included in a multiple-choice item, is best
conducted at the group level. The resulting information is far
more stable in signaling possible areas of concern than examining
incorrect responses of individual students alone. The JMT Data Analysts*[1]
do not recommend the use of individual student by item analysis. For
further information see the question and answer that follow below.
Q8. How helpful is it
to see how an individual student performed on each test question?
A8. It is far more
helpful to review how an individual performed in a specific learning
standard/strand than on a single test question. Student by item data are
very limited. It is important to realize that many factors impact how a
student performs on a single test, including factors that are
situational rather than related to the student’s ability. From a
statistical and psychometric stand point, measurement error for student
performance on individual test items is much greater than for a
composite of items (e.g., a specific standard or strand). Analysis and
interpretation of student performance data should be conducted on the
most reliable index of the student’s ability to avoid misinterpretation
of the results. Scaled Scores are the most reliable index of student
performance on the New York State tests.
JMT
Data Analysts
Kathleen
Maxwell, Capital Region BOCES (kmaxwell@gw.neric.org)
Amy Svirsky,
Capital Region BOCES (asvirsky@gw.neric.org)
Stacy Ward,
HFM BOCES (sward@hfmboces.org)
Jacqueline
Pezzulo, Questar III (jpezzulo@questar.org)
Nicole
Catapano, WSWHE BOCES (ncatapano@wswheboces.org)
Katie Jones,
WSWHE BOCES (kjones@wswheboces.org)
|
