Polio & Friedman Ch 2 for 2/25
What are the advantages and pitfalls of this type of research? What concerns do the authors point out? What attracts you or not about this approach? How could you use these methods to investigate your own topics of interest?
Polio and Friedman talk about the importance of developing studies that exhibit validity, reliability, and practicality. Part of my lean towards qual studies has to do with the issue of reliability in quant studies. In general, researchers creating quant studies in second language writing often default to manipulating an independent variable while the dependent variable is the quality of the text written after the effects of the independent variable have taken place. This is logical as much of what they are trying to quantify is improvement in writing. However, this stems back to the controversy of paths to improvement; if the same definition of improvement isn’t used between one study and the next, the results can’t be compared. In addition, as there’s no consensus on what improvement is, teachers could look at these results with skepticism. This is particularly true for anyone who has taken part in “mass” grading and norming exercises. In ESL Grammar we spend a significant amount of time norming yet still struggle to get people on the same page, and the only way we judge writing is based on grammatical errors, which (in theory) should be more straightforward than writing quality overall. The room for difference just in the grading in this context makes me very skeptical of how the experimenters are quantifying improvement and explaining low interrater reliability. Although I understand the practicality constraints, this seems like a huge issue. In the example study given, I think each piece of writing should have been examined by more than one rater, even if that meant having more raters overall. This is something I will have to keep in mind as I craft my experiment (or try to avoid by doing something more qual-based).
Polio and Friedman talk about the importance of developing studies that exhibit validity, reliability, and practicality. Part of my lean towards qual studies has to do with the issue of reliability in quant studies. In general, researchers creating quant studies in second language writing often default to manipulating an independent variable while the dependent variable is the quality of the text written after the effects of the independent variable have taken place. This is logical as much of what they are trying to quantify is improvement in writing. However, this stems back to the controversy of paths to improvement; if the same definition of improvement isn’t used between one study and the next, the results can’t be compared. In addition, as there’s no consensus on what improvement is, teachers could look at these results with skepticism. This is particularly true for anyone who has taken part in “mass” grading and norming exercises. In ESL Grammar we spend a significant amount of time norming yet still struggle to get people on the same page, and the only way we judge writing is based on grammatical errors, which (in theory) should be more straightforward than writing quality overall. The room for difference just in the grading in this context makes me very skeptical of how the experimenters are quantifying improvement and explaining low interrater reliability. Although I understand the practicality constraints, this seems like a huge issue. In the example study given, I think each piece of writing should have been examined by more than one rater, even if that meant having more raters overall. This is something I will have to keep in mind as I craft my experiment (or try to avoid by doing something more qual-based).
Comments
Post a Comment