SPED 596: Research in Special Education

SPED 596: Research in Special Education. Week 7 (8/05-8/11) Slides adapted from Mertens (2024) and Mertler (2022)..

Presentation. Review Data Collection.

[Audio] Nonexperimental research designs: A group of techniques used to conduct quantitative research where there is no manipulation done to any variable in the study. Reasons for lack of manipulation: The variable was naturally "manipulated" before the study began; It is not possible for the researcher to manipulate that particular variable. Types of nonexperimental research: Descriptive research: observational research and survey research; Correlational research; Causal-comparative research..

[Audio] Purpose of descriptive studies: To describe and interpret the current status of individuals settings conditions or events. Natural study of the phenomenon: The researcher studies the phenomenon of interest as it exists naturally; no attempt is made to manipulate the individuals conditions or events. Two types of descriptive research: Observational research and survey research..

[Audio] Focus: Quantitative observational studies typically focus on a particular aspect of behavior that can be quantified through some measure. Criteria for quantification: Counting of the occurrence of behavior; Determining its overall frequency; The accuracy intensity proficiency or mastery of a particular behavior; An instrument that allows for the rating. Strengths: Yields data that depict the complexity of human behavior Provides a quantitative option to qualitative approaches such as ethnography and grounded theory research. Limitations: Requires considerable advanced planning attention to detail and more time than other descriptive approaches..

[Audio] Survey research: Type of descriptive research focused on describing characteristics of a group or population. Respondents: Sample of individuals selected to respond to a survey. The need for accurate representation: The sample must be selected using a probability sampling technique to ensure more accurate representation of the population. This will avoid the inferences drawn about that population being erroneous to some degree. Investigates relationships between variables: By a combination of survey research and correlational research design. Used in educational and comparative research. A confusion in categorizing survey research as a separate approach to conducting quantitative research or as a particular data collection technique using other approaches to quantitative research. Direct administration and return rate: Direct administration: Method of administering a survey in person to all members of a given group usually at the same time. Return rate: Rate of response to a survey usually expressed as a percentage..

[Audio] Descriptive survey: A one-shot survey for the purpose of simply describing the characteristics of a sample at one point in time. Cross-sectional survey: Survey involving the examination of the characteristics of—and possibly differences among—several samples or populations measured at one point in time. Census: A cross-sectional survey conducted for an entire population as opposed to a sample drawn from the population. Longitudinal survey: Survey where individuals in one group or cohort are studied at different points in time. Trend study: A longitudinal survey study that examines changes within a specifically identified population over time Cohort study: Type of survey study where the researcher studies within a specified population a subgroup (called the "cohort") whose members share some common characteristic. Panel study: Survey study where the researcher examines the exact same people over a specified length of time..

[Audio] Strengths: Enables efficient data collection from a large number of people; Allows for generalizability of results to large populations Is versatile both in terms of what can be investigated and how. Limitations: Low response rates: depends on the nature of the survey study the length of the survey instrument and the population studied. Increased monetary cost of implementing the survey study. Reliance on self-reported data; collecting information on perceptions of what is believed to be accurate; collecting socially acceptable responses..

Sampling Techniques in Quantitative Research (11 Slides).

[Audio] Sampling by quantitative researchers: Quantitative researchers do not typically gather data from an entire population; their intent is to generalize the results of their studies to the entire population. The ability to conclude that the results from their research are applicable to other samples selected from the same population. It is not feasible for data collection and analysis of entire population due to the amount of time required. Sample: correct and accurate: If the sample of potential respondents is selected correctly and accurately the conclusions based on their data should be applicable to the entire population. The key is that the sample must be selected correctly and accurately. Certain characteristics important to the outcome of the study must be taken into consideration so that members of the sample reflect those characteristics in the same manner as the larger population. Initial step: defining the population: The initial step is to define the population to which the researchers plan to generalize their results. Population symbolized in research settings as N is a group of individuals who share the same important characteristics. Members of a population will never share every characteristic but they do share those characteristics that can influence the outcome of the study. Populations can be of any size and can cover almost any geographical area..

[Audio] Target population: It is the group of people to whom the researcher ideally likes to generalize the results of the study. The larger the target population the more important it is to carefully identify an accessible population. Accessible or available population: It is the group from which the researcher can realistically select subjects. population identified for study is a realistic choice in other words an accessible population and not an ideal one in other words a target population. Subset of population: The next step in the process is to select a sample a subset of the population symbolized by n and representative of the accessible population . Quantitative sampling: two types: Quantitative sampling techniques are typically classified into one of two categories: probability sampling techniques and nonprobability sampling techniques..

[Audio] Figure 12.1: Types of Quantitative Sampling Techniques..

[Audio] Chance in probability sampling: A good sample for a quantitative research study is representative of the population from which it was drawn. Though numerous techniques are available for sample selection not all assure the same level of representativeness. Probability sampling techniques permit the researcher to specify the probability or chance that each member of the population will be selected for inclusion in the sample. Random sampling techniques: Probability sampling techniques involves processes of random selection. The sample is chosen in such a way that each member of the population has an equal chance of being selected. Defining characteristics: The chance that each member of the population will be selected for inclusion in the sample can be specified. All members of the population have an equal chance of being selected for inclusion in the sample. If these two characteristics are not met the sampling technique will not be a probability or random sampling technique. Selecting a random sample: five basic procedures: It includes simple random sampling stratified sampling cluster sampling systematic sampling and multistage sampling. Each procedure involves the same basic steps: identify the population determine the appropriate sample size and select the sample..

[Audio] Simple random sampling: The most rigorous form of probability sampling and the best way to obtain a representative sample. This procedure is feasible only if the population from which you are selecting a sample is small. Random samples were once hand-selected using tables of random numbers. Now software programs like Research Randomizer facilitate the development of a table of random numbers and the random selection process. One of the weaknesses of simple random sampling is that representation of specific subgroups is not guaranteed and is purely by chance. Stratified random sampling: It is a process in which certain subgroups often referred to as strata are selected for inclusion in the sample. Strategic selection of participants from various subgroups and is the best sampling technique when a research goal is to compare participants from different groups within the population. There are two types depending on the type of representation. The basic procedures for selecting a sample are the same as for selecting a simple random sample with the exception that the various subgroups must be determined. The researcher must determine whether proportional or equal representation is desired and make appropriate decisions regarding sample size and the process involves the use of a table of random numbers. Proportional random sampling: it is a stratified random sampling process where a sample is selected so the identified subgroups in the sample are represented in the exact same proportion in which they exist in the population. Nonproportional or equal stratified sampling: it is a similar process in which the representations of subgroups in the sample are equivalent to one another as opposed to reflecting the proportions in the population..

[Audio] Cluster random sampling: In some research situations it is not possible to obtain a list of individual members of the population and such a list may not be available or may not exist. Neither simple random sampling nor stratified random sampling can be used as a method of selecting the sample. The best sampling alternative is to select intact existing groups or clusters of participants rather than individuals and this procedure is known as cluster random sampling. Process is similar to simple random sampling with the exception of the sampling unit. In simple random sampling the sampling unit is an individual and in cluster random sampling the sampling unit is an existing group of individual. The advantage is that the population is small and the disadvantage or limitation is that we are less likely to obtain a representative sample of our larger population. Multistage or two stage random sampling: Combination of cluster random sampling and individual random sampling. This process involves a random sample of clusters followed by a random sample of individuals from within the selected clusters. Depending on the size and scope of a research study multistage random sampling can be a more expedient method of randomly selecting participants than simple random sampling. Systematic sampling: Final type of random sampling technique. Every Kth individual in a population list is selected for inclusion in the sample. It is accomplished by calculating something known as the sampling interval. Sampling interval: It is the distance in the population list between each individual selected for inclusion in the sample. K = Population size ÷ Desired sample size..

[Audio] Systematic sampling: disagreement of authors: Authors of educational research textbooks always disagree over whether systematic sampling should be classified as a random procedure or a nonrandom procedure. Its classification depends on the nature of the list of members of the population that the researcher obtains. Systematic sampling: rarely a good option: If the list appears in some sort of preexisting order alphabetical by grade point average by test score then the process will be a nonrandom sampling procedure. Lists typically appear in some sort of predetermined order. Systematic sampling is rarely a good option when trying to obtain a random representative sample. Researcher can take that list and randomize it and use the newly randomized list to select a systematic sample but the process becomes convoluted..

[Audio] Nonrandom sampling techniques: Nonprobability sampling techniques or nonrandom sampling techniques do not permit the researcher to specify the probability that each member of a population will be selected for inclusion in the sample nor do they create a sampling situation where every member of a population has an equal chance of being selected. Four nonrandom sampling techniques: convenience sampling snowball sampling quota sampling and purposive sampling. Convenience sampling: Referred to as accidental sampling or haphazard sampling it is a "targeted" sampling technique whereby the researcher simply studies whoever happens to be available at the time and is willing to participate. Due to the nonrandom and nonrepresentative nature of convenience samples their use should be avoided at all costs. The results and conclusions quantitative research studies that use convenience samples should be questioned. At a minimum studies that use convenience samples should be replicated to verify the findings of the original study and determine whether or not they were a one-time occurrence. Snowball sampling: It is a qualitative sampling technique. It may also be used as a quantitative sampling technique as an alternative to convenience sampling. The researcher relies on current participants to identify other people as potential participants in the study. By using this technique the researcher essentially gives up any control over who makes up the sample in the study. In quantitative study it is a less-than-desirable sampling technique..

[Audio] Quota sampling: It is a variation of convenience sampling. It is a process of selecting a sample based on precise numbers of individuals or groups with specific characteristics. It is used in large-scale survey research when obtaining a list of all the members of the population of interest is not possible. The researchers specify exact characteristics as well as the number of people with those characteristics they need to participate in the study. Individuals who meet the desired qualifications must be readily available and accessible but those who difficult to locate or contact end up being underrepresented. Purposive sampling: People or other sampling units are selected for a particular purpose. Also referred to as judgment sampling because individuals making up the sample are believed to be representative of a given population. Difference between purposive sampling and convenience sampling is that in purposive sampling the researcher clearly identifies criteria for selection whereas in convenience sampling he or she is simply using whoever is available. Does not eliminate the potential for inaccuracy in the resulting sample selection..

[Audio] Constitution of an appropriate sample size: Sample size must be large enough to provide minimally adequate representation of the larger population. Depends upon the nature of the study the research questions guiding the study the specific makeup of the population to which the results are generalized. Sample size determination: rules of thumb: The larger the population size the smaller the percentage of the population required to get a representative sample. Minimum sample size depends upon the type of research conducted. Researchers cite 30 as a minimum guideline for correlational causal-comparative and true experimental research. In correlational research a minimum of 30 participants is needed to establish the existence of a statistical relationship. In causal-comparative and experimental research there should be a minimum of 30 participants in each group. Larger samples improve the likelihood of detecting differences between groups but these group sizes are sometimes difficult to attain. Minimum of 15 participants in each group for causal-comparative and experimental studies with a minimum of two groups. In survey research a minimum of 350 individuals or a minimum of 10% to 20% of the population..

[Audio] Sample size: specific recommendations: For smaller populations N ≤ 100 the entire population should be studied. If the population size is about 500 400 ≤ N ≤ 600 then 50% of the population should be sampled resulting in an n between roughly 200 and 300 participants. If population size is around 1 500 then 20% should be sampled resulting in an n roughly equal to 300. Beyond a certain point N= 5000 the population size becomes almost irrelevant and a sample size of 400 is adequate. Sampling error: There is no guarantee that the composition of the sample will be precisely identical to that of the population. The differences are the result of chance variations within the population. This chance variation over which the researcher has no control is called sampling error and it impacts the results of the study. Sampling bias: It is a systematic sampling error that is generally the fault of the researcher. It occurs when some aspect of the sampling process creates a bias in the data. When a survey researcher gets a return rate of 30% of questionnaires sent out it is important to note that the large number of unreturned surveys may introduce potential bias into the results. If potential bias is severe enough to negatively impact the quality of the data should be noted. The researchers is obligated to disclose the nature of the bias and contextualize the findings and conclusions within those parameters..

Data Collection.

[Audio] 12-1 Explain how operationalizing a concept provides a basis for making decisions about data collection. Operationalizing Concepts The purpose of data collection is to learn something about people or things. The focus is on the particular attribute or quality of the person or setting. The concept of youth resilience is important to both educators and psychologists who are interested in supporting positive development. The conceptual definition of resilience that emerged was "doing well despite adversity." Four clusters of resilience concepts that "reflect individual (for example assertiveness problem-solving ability) relational (for example social competence quality of parental monitoring) community (for example rites of passage safety and security) and cultural (for example affiliation with a religious organization a life philosophy) aspects of resilience." In the initial stages of planning data collection there are two challenges to the researcher: First the attributes of interest must be identified. Second a decision must be made about how to collect data about those attributes. The process of determining what to collect data about and how to do it is often referred to as operationalizing. The researcher identifies the strategies that will make it possible to test the concepts and theories posed through the research question. Ungar and Liebenberg (2011) used a complex process to develop an instrument that would allow them to measure or operationalize the concept of resilience. The concept of resilience was operationalized in the form of the Child and Youth Resilience Measure (C-Y-R-M-). The C-Y-R-M consists of 58 questions that are asked at all study locations; each location could add up to 15 unique questions if the people in each country chose to do so. The 58 questions encapsulated the concept of resilience as it was expressed by the members of the diverse cultural groups. Tests are an important part of data collection in education and psychology. One reason is that tests are used for hypothesis testing..

[Audio] 12-2 Identify the characteristics of multiple types of quantitative measures including standardized and nonstandardized testing norm-referenced and criterion-referenced testing individual and group tests speed and power tests performance and portfolio assessment curriculum-based assessment and secondary sources of data. Quantitative Measurement Topics A researcher needs to decide whether to (a) use a measurement instrument that is commercially available or one developed by other researchers (b) adapt an existing instrument or (c) create a new one to meet the needs of the proposed research..

[Audio] 12-2 Identify the characteristics of multiple types of quantitative measures including standardized and nonstandardized testing norm-referenced and criterion-referenced testing individual and group tests speed and power tests performance and portfolio assessment curriculum-based assessment and secondary sources of data. Quantitative Measurement Topics Data Collection and Technology Specific uses include the administration of assessment on computer and the use of simulations in performance assessment the use of artificial intelligence to track the test taker's response and the offering of new types of accessibility. Technology supports using simulated performance tasks and allowing for multiple modes of responding. The complex scoring rubrics and algorithms might produce a lower score for a test taker who uses an accommodation because of a disability. Although the use of process data is not common in reporting the process of answering is important when accommodations are made. Fairness needs to be considered especially for individuals with disabilities who might be disadvantaged when deficiencies in skills or use of the accommodation could affect their responses. Welch and Dunbar warn that the quick unexamined acceptance of new item formats with very little research might affect students' scores or change measurement of student achievement. The use of technology has an added importance because of the growth in the area and also because of the use of remote testing in COVID-19. A number of innovations in data collection are possible because of advances in technology that can improve assessment. The use of technology has become pervasive in data collection because of the use of mobile technologies as well..

[Audio] 12-2 Identify the characteristics of multiple types of quantitative measures including standardized and nonstandardized testing norm-referenced and criterion-referenced testing individual and group tests speed and power tests performance and portfolio assessment curriculum-based assessment and secondary sources of data. Quantitative Measurement Topics Performance and Portfolio Assessment Performance assessment is a process for collecting information through systematic observation in order to make decisions about an individual. Performance assessment relies on the use of multiple types of assessments not a single test or measurement device and assessment occurs across time. The primary vehicle for assessment is the direct observation of performance in the form of behavior and products. Performance assessment is an essential element of alternative assessment and the portfolio is the vehicle through which performance assessment information is stored. Portfolios can be used as a method of data collection in research. With appropriate ethical clearance the student work in a portfolio can serve dual purposes: teacher assessment and research data. Morrow and others (2018) defined portfolios as purposeful systematic and meaningful collections of student work that are evaluated and measured against predetermined scoring criteria. There are challenges to using portfolios in research and evaluation studies because of the subjective nature of the collection and scoring of the information found in the portfolio. Issues that need to be considered: How the content will be selected for inclusion in the portfolio. What quality of work will be included—best versus typical. Whether students should participate in selecting the work that will be included. How much information is required to get a true score. How reliability and validity of the information will be determined. One method for scoring the information in portfolios is to use a rubric. A rubric is a scoring tool for verbally describing and scaling levels of student achievement as represented for example by products found in a portfolio. The rubric presents a gradation of performance with various types of anchors: poor to excellent or above standard at standard and below standard. The developmental rubric is the most useful because the score provides a gauge for where the participant is on a continuum and allows the researcher to make comparisons that are both criterion referenced and normative. The 4teachers website is a good resource for the integration of technology for portfolios and rubrics. Tips for developing rubrics based on ideas found at teachersfirst.com: Look for already developed rubrics and see if you can use those or adapt them to your purpose. Ensure that the rubric addresses the most important aspects of student performance (in other words check the validity of the rubric). Include between 3 and 6 gradations (for example from excellent to needs improvement). Make sure the rubric is clear and descriptive so raters and students understand what constitutes the various levels of performance. Try the rubric out on some actual samples of student work. See if you and your colleagues can usually arrive at consensus about what scores to assign a piece of student work (in other words check the rubric's reliability)..

[Audio] 12-2 Identify the characteristics of multiple types of quantitative measures including standardized and nonstandardized testing norm-referenced and criterion-referenced testing individual and group tests speed and power tests performance and portfolio assessment curriculum-based assessment and secondary sources of data. Quantitative Measurement Topics Testing and Technology Increased access to computers has coincided with an increase in the collection of test data using this technology. The advantages of using computers for this purpose are clear: ease and flexibility of administration and grading of tests. An important question for researchers relates to the equivalency of computer-based testing and paper-and-pencil testing. The Educational Testing Service identified several ways that technology can enhance data collection: Using computer tablets to assess English language competency for people who are not proficient in English. Providing tracking for progress on measures in longitudinal studies. Designing items that adapt to the user such as traditional keyboard or touch screen. The National Center for Fair and Open Testing maintains a list of problems when computer examinations are used. They caution that use of computer examinations should not be used in high-stakes testing situations because of the ongoing failures that they have documented. Data collection using computers offers many opportunities for innovations. Computers can be used to present simulations as assessment tools and then they can capture the performance in intricate detail. Computers can be used to record each movement that a participant makes which allows the researcher to analyze strategies points of difficulty and thinking processes..

[Audio] 12-2 Identify the characteristics of multiple types of quantitative measures including standardized and nonstandardized testing norm-referenced and criterion-referenced testing individual and group tests speed and power tests performance and portfolio assessment curriculum-based assessment and secondary sources of data. Quantitative Measurement Topics Secondary Data Sources Secondary sources of data can be found in sources such as administrative records previous research studies and extant databases. There are several quantitative databases: the National Assessment of Educational Progress (N-A-E-P-) the High School and Beyond Longitudinal Study the National Longitudinal Transition Study of Special Education Students the National Center for Education Statistics and the Schools and Staffing Survey (S-A-S-S-) as well as a newer version called the National Teacher and Principal Survey. Such databases can provide a wealth of information; but they should be used with appropriate caution. Some national databases do not allow for disaggregation by variables such as race gender or type of disability. In others certain categories of persons have actually been excluded. Access to secondary data in the form of big data is one of the newest concepts in data collection. Big data are data that may or may not have been produced for a specific research project and are characterized by high volume high velocity and high variety. The data can come from multiple sources including government agencies and individual computer use internet use and other forms of technology such as smartphones. The use of big data comes with challenges: The researcher does not control the quality of the data the relevance of the data or the feasibility of merging data sets from different sources. Ethical issues arise regarding use of geographic or pictorial data that might have been collected without each person's consent..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Selection and Development of Quantitative Instruments Identifying an Instrument: Sources of Information The first source of information about possible data collection instruments for research is the literature review. Important sources of information about measurement instruments: The Mental Measurements Yearbooks (M-M-Y--) are published by the Buros Center for Testing at and contain descriptive information on hundreds of tests in many major categories. Tests in Print (T-I-P--) also from the Buros Center for Testing is a volume that serves as a supplemental source of information to the M-M-Y in that it contains a comprehensive bibliography of all tests that appeared in preceding M-M-Y--. it contains information about any test that is in print and available for purchase or use. PRO-ED Inc. publishes tests and test critiques of standardized tests in the areas of speech-language pathology special education and rehabilitation psychology and counseling occupational and physical therapy and early childhood. The A-P-A provides information on selecting and ethical use of both published and unpublished tests. The Educational Testing Service (E-T-S--) provides a wide range of means to access information about tests. It maintains a test collection that contains thousands of tests and other measurement devices from the United States and a few other countries (Canada England and Australia)..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Selection and Development of Quantitative Instruments Advice for Selecting an Instrument Things to consider to help in selecting a data collection instrument: Define the purpose for testing the content and skills to be tested and the intended test takers. What are the variables that the test measures? What subscales are included in the test? Review and select tests based on the appropriateness of test content skills tested and content coverage for the intended purpose of testing. Review materials provided by test developers and select tests for which clear accurate and complete information is provided. Select tests through a process that includes persons with appropriate knowledge skills and training. Evaluate evidence of the technical quality of the test provided by the test developer and any independent reviewers. Evaluate representative samples of test questions or practice tests directions answer sheets manuals and score reports before selecting a test. Evaluate procedures and materials used by test developers as well as the resulting test to ensure that potentially offensive content or language is avoided. Select tests with appropriately modified forms or administration procedures for test takers with disabilities who need special accommodations. Evaluate the available evidence on the performance of test takers of diverse subgroups. Questions to ask if modifications are needed in an available instrument: If accommodation is made on the basis of a specific characteristic (for example a disability) how should eligibility for accommodation be determined? What type of modifications should be allowed? Do scores achieved under nonstandard conditions have the same meaning? If there is a difference in performance levels between standard and nonstandard administrations are these due to actual differences in the construct being measured or are they artifacts of modifications of the testing process? Researchers can use pilot tests to determine the impact of modifications of existing instruments..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Development of a data collection instrument is a complex and time-consuming task. Step 1. Define the Objective of Your Instrument Bernard and others set out to develop an instrument that provides a multidimensional approach to measuring individual differences of 15 human motives. They worked with an evolutionary psychological theoretical framework: "motivated behavior is purposeful behavior. Purposeful behavior is neither random nor reflexive. Purposeful behavior solves challenges to survival posed by the 'Environments of Evolutionary Adaptedness' (E-E-A--).".

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Step 2. Identify the Intended Respondents and Make Format Decisions Relevance of criterion centers on the persons for whom the measurement is intended. Factors related to the administration of the instrument: Time required to complete it. Reading level. Format for items. Response option formats. Test setting. Format options: True-false. Matching. Multiple choice. Sentence completion. Ranking items. Likert-type scales. Open-ended essay-type questions. Being able to recognize an item type and being able to write good high-quality items are two different matters. The Likert-type scale is the type of item that makes a statement. EX: "My emotional problems interfere with my usual daily activities." Then the test taker would indicate the strength of their agreement or disagreement with the statement on a four or five point scale: 1 = strongly agree 2 = moderately agree 3 = neutral 4 = moderately disagree 5 = strongly disagree..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Step 3. Review Existing Measures Bernard and others (2008) use their extensive review of literature on motivation to derive a list of 15 motives each related to a social domain: Self-Protection Domain—Aggression Curiosity Health Play and Safety. Mating Domain—Sex and the "status motives" of Appearance Material Mental and Physical. Relationship Maintenance/Parental Care Domain—Affection. Coalition Formation Domain—Altruism and Conscience. Mimetic Domain—Legacy and Meaning..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Step 4. Develop an Item Pool There are many avenues for the researcher in preparing draft items for a new measurement device. Some may be adopted or adapted from current measures. Others might be developed using experts or program staff responsible for the program being studied. Cultural Responsiveness in Instrument Development Important advice about collecting data in diverse multicultural contexts: ensure that members of the targeted community are included in the review process of the constructs procedures and instruments. Resources that have addressed concerns about multicultural issues in the construction of data collection instruments: Multicultural Guidelines: An Ecological Approach to Context Identity and Intersectionality. Ethnicity Race and Cultural Affairs. Guidelines on Race and Ethnicity in Psychology: Promoting Responsiveness and Equity. Guidelines for Psychological Assessment and Evaluation. Guidelines for Assessment and Intervention With Persons With Disabilities. Guidelines for Psychological Practice with Sexual Minority Persons. Standards for Educational and Psychological Testing. The Committee on Impacts of Sexual Harassment in Academia's Sexual Harassment of Women: Climate Culture and Consequences in Academic Sciences Engineering and Medicine. It is important to take time to think of an appropriate title for the data collection instrument. Include a short introductory paragraph after the title to explain the purpose of the instrument and its intended use. Directions on how to complete the instrument should be included after the intro paragraph..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Step 5. Prepare and Pilot Test the Prototype The researcher will assemble the first draft of the instrument after the item pool has been developed. It is recommended that the developer review the literature and ask other professionals and community members who are knowledgeable about the attribute and its measurement in the targeted sample to review the prototype. After revisions have been made the prototype can be tried out on a small sample of the intended respondents. The researcher should provide a means for the members of the pilot group to give feedback on the instrument in terms of items that might need additional clarification. The final pilot test should be conducted with a large enough sample to enable the researcher to gather reliability and validity information. If the instrument depends on the use of interviewers observers or document reviewers the researcher must collect interrater and intrarater reliability indices at this time..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Step 6. Conduct an Item Analysis and Revise the Measure The answers to each item should be reviewed to determine if a pattern suggests ambiguity or bias in the item. It is good to interview the pilot participants to gather information about their reactions to the various items (if logistics permit it). The researcher should be careful to document all the pilot test procedures and revisions in the instrument so that these can be presented in the research report as evidence of the quality of the measurement..

[Audio] 12-3 Describe and apply steps for the selection and development of data collection instruments including sources of previously developed instruments and steps in the development of your own instrument. Developing a Data Collection Instrument Advice for Administering Data Collection Instruments A-P-A has several documents that provide advice about administering instruments: Guidelines for Psychological Assessment and Evaluation (APA 2020) and the Standards for Educational and Psychological Testing. Advice on administering instruments for data collection (provided by the Standards): Follow established procedures for administering tests in a standardized manner. Provide and document appropriate procedures for test takers with disabilities who need special accommodations or those with diverse linguistic backgrounds. Provide a reasonably comfortable environment with minimal distractions. Provide test takers with an opportunity to become familiar with test question formats instructions and any materials or equipment that may be used during testing. Protect the security of test materials including respecting copyrights and eliminating opportunities for test takers to obtain scores by fraudulent means. If test scoring is the responsibility of the test user provide adequate training to scorers and ensure and monitor the accuracy of the scoring process. Develop and implement procedures for ensuring the confidentiality of scores..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods The researcher is the instrument in qualitative research studies. The researcher is the instrument that collects data by observing interviewing and examining records documents and other artifacts in the research setting or by using some combination of these methods. Three main qualitative data collection methods: observation; interview; and document records and artifacts review..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Qualitative Data Collection and Mobile Devices The Qualitative Report (T-Q-R--) is a website maintained by the Nova Southeastern University that contains many resources relevant for qualitative researchers. The website also includes a section dedicated to mobile research applications that is an annotated list with live links to many apps that can be used for the collection of qualitative (and quantitative and mixed) data..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Observation Qualitative observations differ based on the specific approach and purpose of the study as well as the beliefs of the researcher and the demands of those being observed. Different roles of the observer: Complete observer. Observer-as-participant. Participant-as-observer. Complete participant. Observation at a distance. Idea for an observer to attend to while observing: Program Setting. Human and Social Environment. Program Activities and Participant Behaviors. Informal Interactions and Unplanned Activities. Attend to the Native Language of the Program Participants. Nonverbal Communication. Unobtrusive Measures. Observing What Does Not Happen..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Group and Individual Interviews and Technology Not all researchers have the opportunity or inclination to conduct extensive observations. Qualitative researchers almost always include interviewing as an important method of data collection. Interviews can be structured or unstructured in person or via electronic means in groups or with an individual. Interviewing can be conducted as a part of participant observation or even as a casual conversation. Individual Interviews Hesse-Biber (2017) identifies a continuum of types of interviews from formal to informal with varying degrees of structure. Qualitative researchers tend to favor semi-structured or unstructured individual interview formats. Using very general open-ended questions allows the respondent's concerns and interests to surface providing a broader lens for the researcher's gaze. A more structured approach to an interview means that the researcher develops an interview guide with topics issues or questions that they intend to cover during the interview. Increasing the degree of structure has both advantages and disadvantages. It can help ensure the relevant topics are addressed in the interview but it also must be used with caution to allow the participant's voice to be expressed without undue pressure from the interviewer. Feminists have endorsed the idea that interviewers should be cognizant of power differences between the interviewer and the participants. The focus is on trying to uncover knowledge about women's lives that reflects the diversity of experiences as well as illuminates hidden aspects of their lives related to social justice. Interviewing people with disabilities can present challenges because of the abilities or communication needs of the respondents. Different types and severity of disabilities require entirely different strategies for interviewing. Researchers who work in ethnic/racial minority and Indigenous communities have presented interviews as opportunities to produce a counter-narrative or counter-storytelling. Counter-narratives or -stories are narratives that capture divergent views contrasting the perspectives of the powerful and the marginalized. This strategy of data collection is used to make visible the subtleties of racism and how these beliefs influence life chances institutions and relationships..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Interviews and Technology Researchers with smart phones can use apps to record audio and video. Recording Interviews In some circumstances researchers choose to record their interview in others they must record their interviews. Ethical concerns direct that researchers explicitly turn control of the interview over to the person being interviewed. Turning over control means allowing the person to end the interview at any time choose not to answer specific questions raise issues that the researcher did not bring up and have the opportunity to review their comments before they are made part of the official record for the research data. Discussion Board Qualitative Data Collection Online group interviews can be conducted using discussion boards that are prevalent in educational software used in classrooms. This type of data has advantages in that participants from many locations can be involved at times that are convenient to them. Data on a discussion board are already in text format so there is no need for transcription of the data. Discussion boards do have their disadvantages though: Discussion board participants can go off topic. They can read others' comments and not contribute their own. They can be slow because of their asynchronous nature. Some participants may dominate the discussion. Respondents may have difficulty because they are unfamiliar with the technology or their equipment or browser may be old and incompatible with the research requirements. Strategies to address these challenges include having assistants who can assist respondents who are having trouble. Researchers might also have to modify the instrument itself so that it can be displayed properly on older equipment or in different browsers. Focus Groups Focus groups can be viewed as a data collection method or as a strategy for research. Focus groups are group interviews that rely not on a question-and-answer format of interview but on the interaction within the group. Focus groups can be conducted in person or through technology using software such as FocusGroupIt (www.focusgroupit.com) or Zoom (zoom). Using focus groups as a research strategy would be appropriate when the researcher is interested in how individuals form a schema or perspective of a problem. Systematic variation across groups is the key to research design with focus groups. Examples include composing groups that vary on different dimensions: Variation in terms of characteristics such as age ethnicity gender or disability. Using homogeneous groups versus heterogeneous groups. Warning: Hostility can result from bringing together two groups whose lifestyles do not normally lead them to discuss a topic together. Comparing responses of individuals who are brought back for more than one group (in other words the same group meets several times together). The determining criteria for group composition vary with the purpose of the research. The group is considered the unit of analysis; therefore the researcher must decide how many groups to have. A "rule of thumb" suggests starting with four to six groups and adjusting that number to meet specific research questions. Rodriguez and others (2011) make the argument that focus groups provide a potential mechanism suited to the advancement of an agenda for social justice. Focus groups can be used in needs sensing for training and service programs for instrument review and for other research purposes. Things to consider when constructing questions for focus groups: Include fewer than 10 questions and often around 5 or 6 total. Use open-ended questions. Avoid using "why" questions. Brainstorm sessions with colleagues to generate questions. Begin by establishing the context for questions so participants are ready to respond. Arrange questions in a logical order sometimes from general to specific. Example of Focus Group Questions Gregg and others (2012) used these questions (as well as others) with follow-up probes in their focus group study that examined Latino families' involvement in their child's early education: Tell me about your child's strengths and interests. Probe: Tell me about opportunities you have to share information about your child's strengths and interests with your child's teacher. Tell me about the hopes and dreams you have for your child. Probe: Tell me about the opportunities you have to share your hopes and dreams for your child with your child's teacher. How do you communicate with your child's teachers and family partners? The facilitator needs to be able to control the interview process so that all participants can express themselves one or a few people do not dominate the discussion more introverted people are encouraged to speak and all important topics are covered..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Document and Records Review and Technology All organizations leave trails composed of documents and records that trace their history and current status. Documents and records include not only the typical paper products such as memos reports and plans but also computer files and other artifacts. Documents and records give the researcher access to information that would otherwise be unavailable. The researcher needs to be aware of the origin and purpose of the documents. Records or documents can be prepared for either official or personal reasons. A second distinction that is important for documents/records review is the source of the materials. Qualitative researchers need to guide their choices of documents and other materials by asking how the evidence in the materials fits with what is found in other types of data. Examining patterns and inconsistencies in the evidence is a way to judge the value of the evidence in the documents. Criteria to consider when conducting a document review search: Authenticity. Credibility. Meaning. Representativeness..

[Audio] 12-4 Identify the characteristics of qualitative approaches to data collection including observations individual and focus group interviews records and document review participatory data collection strategies and visual data. Qualitative Data Collection Methods Visual Data and Technology Visual data include photographs film video painting drawing collage sculpture artwork graffiti advertising cartoons maps graphical displays doodling and other types of images. Visual data can be produced by the researcher produced by participants in the research study or extant. Tiidenberg (2018) examined ethical issues involved in the collection of visual data including photographs and videos especially as it relates to privacy issues. Criteria to consider when making a decision about the use of digital visual data: What are people's expectations of privacy? How vulnerable are the people depicted in the visual images? How sure can you be that you are not causing harm by displaying the images as part of your research data? Visual data collection strategies: photography (photovoice and photo elicitation) and G-I-S-. Photovoice: a data collection method in which the participants are asked to take pictures as a part of the research study itself. Photo Elicitation: the use of visual images to generate verbal discussions as a part of qualitative data collection. G-I-S (geographic information systems): mapping practices of spatial analysis and cartographic displays that can present geographic data that enhance understandings through layered visual displays..

[Audio] 12-5 Discuss implication of different strategies for data collection including technology mixed methods approaches and Indigenous approaches. Qualitative Data Collection Methods Participatory Data Collection Strategies Participatory data collection strategies tend to fit more comfortably into the qualitative approaches although not all qualitative approaches are participatory. Participatory Rural Appraisal P-R-A includes a number of different techniques all of which are based on some version of semi-structured interviewing that are aimed at sharing learning between local people and outsiders. Mapping Techniques: Various forms: (a) historical maps; (b) social maps; (c) personal maps; or (d) institutional maps. Ranking Exercises: Exercises can be done to rank such things as problems preferences or wealth. Trend Analysis: P-R-A researchers have undertaken trend analysis using such things as seasonal calendars and daily activity charts. Beneficiary Assessment A beneficiary assessment (B-A---) is conducted to determine the perceptions of the beneficiaries regarding a project or policy. Three methods of data collection are used in BA: semi-structured interviews focus group discussions and participant observation. Self-Esteem Associative Strength Resourcefulness Action Planning and Responsibility The creative techniques that can be used in a sarar include mapping and nonserial posters to encourage participants to reflect on their lives and experiences. Nonserial posters involve the use of poster-sized pictures that depict dramatic human situations that are fairly ambiguous in nature. Investigative techniques such as a pocket chart can also be used. The pocket chart is a matrix made of pockets that are labeled by simple drawings. Gender analysis is a technique that gives insights into differentiated impact on the access to and control of domestic and community resources. Several planning techniques can be used in sarar: "Story with a gap " in which one picture depicts "before" and the second "after " and participants are encouraged to discuss the steps needed to reach the situation in the "after" picture. Force-field analysis in which "before" and "after" scenarios are used and the participants are encouraged to identify what factors will facilitate/inhibit the achievement of the desired picture. Appreciative Inquiry Appreciative inquiry (A-I---) looks at organizational issues challenges and concerns by focusing on what is working particularly well in an organization. (A-I ) typically has a four-step process: Inquiry Phase in which participants interview each other sharing their peak experiences in the organization with another individual and then with the group. Imagine Phase asks participants to envision themselves and their organization working at their best. Innovate Phase participants use their dreams to propose strategies processes and systems that will help create and support positive change. Implement Phase participants begin the implementation of the identified strategies from the Imagine and Innovate Phases. An example of (A-I ) was based on an evaluation of an early childhood education program in the time of the COVID-19 pandemic. The use of (A-I ) allowed the researchers to track shifts in perceptions of what was important and what was working well with the pandemic required changes in practices. The researcher must fulfill the role of educator to prepare the participants to engage in the processes. The key to success is to be flexible and innovative and willing to adapt to the local circumstances..

[Audio] 12-5 Discuss implication of different strategies for data collection including technology mixed methods approaches and Indigenous approaches. Qualitative Data Collection Methods Indigenous Data Collection Indigenous researchers are free to use many kinds of data collection strategies. They do so with the critical lens of ensuring that they are culturally responsive to the needs of the group they are researching with. Indigenous data collectors also need to observe the appropriate cultural protocols about "what can be said and written how it can be said and written where it can be written by whom and for whom.".

[Audio] 12-5 Discuss implication of different strategies for data collection including technology mixed methods approaches and Indigenous approaches. Mixed Methods and Data Collection Different designs reflect different data collection strategies such as collecting quantitative data first to use as a basis for collecting more in-depth qualitative data. Data collection decisions in mixed methods research should be guided by the purpose of the research and the belief systems of the research team. The strength of mixing data collection methods is that one can capitalize on the strengths of one method over another for a particular purpose. Mixed methods research is fertile ground for research on the implications of different data collection choices. Concept mapping is a mixed methods approach that is used in program planning development and evaluation. It combines qualitative data collection strategies such as focus groups with the use of concept mapping software that quantitatively analyzes the qualitative data. Concept mapping consists of six steps: Preparation. Generation of statements. Structuring of statements. Representation of statements. Interpretation of maps. Utilization of maps. The World Bank's (2013a) RES-360 (resilience) data collection strategy is another example of mixed methods. It was designed as a rapid assessment tool to assess risks and resilience in education systems particularly in contexts of violence and conflict. Social network analysis is another example of a mixed methods data collection strategy. Data collection using social network analysis involves asking a group of individuals to define their relationships in friendship work consultation and other situations in which collaboration occurs..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection The way that the researcher chooses to operationalize the attributes is crucial in that this determines the inferences that can be made from the data. The researcher needs to consider the quality of the data collection strategy. The researcher must establish indicators that provide evidence that the information generated in the research is trustworthy and believable. Three standards for judging the quality of quantitative research measurement: reliability validity and objectivity..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Postpositivist: Reliability Data collection instruments must be consistent to be useful. The more reliable the measurement the better the researcher can arrive at a true estimate of the attribute that the instrument purports to measure. The purpose of measurement is to get an accurate estimate of a particular attribute. Accuracy is achieved by minimizing sources of error as much as possible and obtaining an estimate of how much error remains. Two types of error can influence performance on a measurement instrument: systematic and unsystematic. Systematic errors inflate or deflate performance in a fixed way and thus do not affect a measure's reliability. Unsystematic errors vary at random from situation to situation and therefore cannot be predicted. Unsystematic errors fall into three categories: Those within the person being measured. The conditions of the administration of the measurement. Changes in the measurement instrument or tasks. Reliability is calculated using a statistic that compares performances by the same individuals at different times or on different parts of the instrument. The reliability coefficient is interpreted much like a correlation coefficient. Researchers can use several approaches to determine the reliability of a particular data collection instrument. Two common approaches: repeated measures and calculation of internal consistency. Repeated Measures Reliability Two types of repeated measures reliability are the coefficient of stability and the alternate-form coefficient. Coefficient of Stability (Test–Retest): Involves administering a test to a group of individuals waiting a period of time and then administering the same test to the same individuals a second time. One of the drawbacks of this approach is the potential for practice effects or remembering items across administrations of the test. Alternate-Form Coefficient (Parallel Forms): An equivalent form of the test is used in the second administration. In addition to eliminating the practice effect this approach enables the researcher to determine the degree to which performance might be influenced by new items. The major concern with the parallel-forms reliability check is the degree to which the tests are equivalent. Internal Consistency The method of internal consistency can be used with only one administration of an instrument. It is appropriate for use when the instrument has been designed to measure a particular attribute that is expected to manifest a high degree of internal consistency. The most frequently used procedures are Cronbach's coefficient alpha and various Kuder-Richardson formulas. These formulas can be used to compare responses within one administration of an instrument to determine its internal consistency. Reliability with Observers Interrater Reliability: the reliability between two independent observers or raters; can be expressed either as a reliability coefficient calculated between two sets of observations collected by independent observers or as a simple percentage of agreement between the two observational data sets. Intrarater Reliability: the comparisons are made between two data collection efforts by the same observer. The calculations are the same: either a reliability coefficient or a percentage of agreement..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Constructivist: Dependability The quality of the data collection can be determined by means of a dependability audit. The change process can be inspected to attest to the quality and appropriateness of the inquiry process..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Postpositivist: A Unified Concept of Validity The conventional definition of the validity of an instrument is the extent to which it measures what it was intended to measure. The validity of an instrument is assessed in relation to the extent to which evidence supports the interpretations of test (or instrument) scores for the proposed uses of the test. A concern related to validity: the extent to which the instrument measures the attributes it was intended to measure rather than bias due to gender race and ethnicity class disability or other cultural factors. The Standards identify two threats to validity: Construct underrepresentation. Construct-irrelevant variance. Sources of Evidence for Construct Validity Validity is a unified concept and that multiple sources of evidence are needed to support the meaning of scores. Researchers interested in measuring a hypothetical construct need to explicate the theoretical model that underlies the constructs. Controversy frequently surrounds valid interpretations of test scores in measurement because of cultural differences that can result in different performances. Evidence Based on Response Patterns Researchers can use a variety of strategies to determine if their interpretation of scores is valid for a given construct. One strategy is to examine response processes of the test takers to see if there is a match between the processes they are using and the construct of the test. They can also explore differences in the test takers' interpretation of the items. Evidence Based on Relationship of Test Scores with Other Variables Evidence of validity can also be established by examining the relationship of the test scores to another variable that the theory suggests should hold. Correlation with other tests that measure the attribute of interest or by means of factor analysis to support the structure of the attribute as it has been designed by the researcher. Borsboom (2005) cautions that correlations between tests do not in and of themselves reveal validity. What it really means for a test to measure a psychological attribute is "the primary objective of validation research is not to establish that the correlations go in the right directions but to offer a theoretical explanation of the processes that lead up to the measurement outcomes." Evidence Based on the Internal Structure of the Test Other statistical strategies can be used as evidence of validity such as factor analysis and hierarchical linear regression analysis. These strategies reveal the internal structure of the test. Test makers might intend to have their tests measure a single unitary concept or several concepts. Additional research using confirmatory factor analysis is needed to establish the validity of the measure and possible subscales..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Postpositivist: A Unified Concept of Validity Evidence Based on Test Content Kettler and Dembitzer (2022) describe content validity as "the degree to which the mainifest content of a test including its administration accurately reflects the underlying construct it is meant to measure." Evidence for content validity can be obtained by systematically analyzing the alignment between items and content standards and/or by expert review of the content of the times and the tests. The researcher needs to be certain that the test covers the appropriate content. Validity evidence of the content of a test and the construct it is intended to measure can be explored by reviewing the items or tasks in the measurement instrument to determine the degree to which they represent the sample. It is helpful to build a specifications matrix that lists the items and the content area domains covered by each item. The higher the degree of overlap the better the representation of the behavioral domain and the stronger the evidence for validity. Validity evidence regarding the content of a test can also be obtained by using content experts to make judgments following a process based on alignment methodology. Evidence of validity with regard to content is especially important in studies that purport to compare two (or more) different curricula teaching strategies or school placements. Evidence Based on Test—Criterion Relationship Tests such as the G-R-E or the M-C-A-T are used by universities to predict who will be successful in their programs. The Standards describe two designs associated with test–criterion relationships: predictive studies that examine the strength of relationship between test scores; the criterion at a later time and concurrent designs that establish the relationship between the test scores and the criterion at the same time. Be aware of a number of variables that can constrict the value of a coefficient that is used as evidence of a predictive relationship. Problems with the G-R-E approach: Universities try to be selective as to whom they admit so they are not likely to admit everyone regardless of G-R-E scores. Typically universities use more than one criterion for selecting students so G-R-E scores are not the only basis for selection. The students who are accepted probably represent a restricted range on the predictive variable. Students who actually complete the program will also represent a restricted range of values on the criterion variable because generally graduate students can make only As or Bs. Many personal factors influence a student's performance in graduate school such as motivation economic conditions and family responsibilities. Evidence for Validity and the Consequences of Testing Researchers need to be aware of the positive and negative consequences of testing that go beyond the interpretation of the test scores themselves. Tests are given in complex contexts and decisions are made on the basis of tests that impact across the systems. Researchers need to gather evidence of the intended and unintended consequences of testing to establish that the claimed benefits of testing are occurring and that negative consequences are not. Gathering evidence about the consequences of using tests and making decisions based on the test scores is an important area for researchers to consider..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Constructivist: Credibility Guba and Lincoln (1989) identify credibility as the interpretive parallel to validity. Research strategies that can enhance credibility: Prolonged and substantial engagement. Persistent observation. Peer debriefing. Negative case analysis. Progressive subjectivity. Member checks. Triangulation..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Postpositivist: Objectivity Objectivity refers to how much the measurement instrument is open to influence by the beliefs and biases of the individuals who administer score or interpret it. Objectivity is determined by the amount of judgment that is called for in these three processes. More objective measures consist of short-answer multiple-choice and true-false format options. Less objective measures include essay tests although these can be made more objective by establishing criteria for scoring the responses. Objectivity is deliberately sacrificed to allow the respondents a wider range of possible responses (in some cases). Objectivity in projective tests can be exchanged for the value of having expert judgment involved in the administration scoring and interpretation of the test results..

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Constructivist: Confirmability A confirmability audit can be used to trace the data to their original sources and to confirm the process of synthesizing data to reach conclusions using a chain of evidence. Critics of the use of observation as a data collection strategy often raise the question of observer bias. Constructivists argue that the closeness of the researcher who develops appropriate relationships in the field yields more valid data than observations conducted by a distant and detached researcher. Wasterfors (2018) argues for direct observation as a way to avoid bias that is associated with "armchair research.".

[Audio] 12-6 Describe the meaning of standards for judging the quality of data collection from the vantage point of various paradigms including reliability and dependability; validity and credibility; objectivity and confirmability; and avoidance of bias based on dimensions of diversity such as gender race and ethnicity disability or Indigeneity. Standards for Judging Quality of Data Collection Transformative Paradigm: Avoidance of Bias Four standards for fairness: Cluster 1. Test design development administration and scoring procedures that minimize barriers to valid score interpretations for the widest possible range of individuals and relevant subgroups. Cluster 2. Validity of test score interpretations for intended uses for the intended examinee population. Cluster 3. Accommodations to remove construct-irrelevant barriers and support valid interpretations for scores for their intended uses. Cluster 4. Safeguards against inappropriate score interpretations for intended uses. Two issues that are cross-cutting: One is who has the power to say what is on the test and the methods used for administration. The second is the concept of intersectionality. Writings about data collection from the perspectives of feminists people with disabilities racial and ethnic minorities (African American and Latino American) Indigenous people and English learners. Feminist and Sexual Minorities and Data Collection. People With Disabilities: Issues in Data Collection. Racial Ethnic and Language Minorities: Issues in Data Collection. Strategies for test accommodation: (a) modification of presentation format; (b) modification of response format; (c) modification of timing; (d) modification of test setting; (e) using only parts of a test; (f) using alternate assessments. There are situations when test accommodations may be inappropriate. The test taker's disability may be directly relevant to the characteristic being measured in the assessment. A second challenge to the appropriateness of modification occurs when the purpose of the assessment is to identify a specific disability..

[Audio] 12-7 Describe the application of the principles of Universal Design for Learning (U-D-L--). Standards for Judging Quality of Data Collection Universal Design and Instrument Development The Every Student Succeeds Act (2015) requires that all K–12 assessments be developed using the principles of Universal Design for Learning (U-D-L--). The Center for Universal Design defined universal design as "the design of products and environments to be usable by all people to the greatest extent possible without the need for adaptation or specialized design." U S federal legislation defines U-D-L as a framework that "(A) provides flexibility in the ways information is presented; and (B) reduces barriers in instruction provides appropriate accommodations supports and challenges and maintains high achievement expectations for all students." The Center for Applied Special Technology developed guidelines for U-D-L that are based on three principles: Principle I Provide multiple means of representation. Principle II. Provide multiple means of action and expression. Principle I-I-I--. Provide multiple means of engagement. Specific problems related to the use of standardized tests with racial and ethnic minorities have been identified by a number of authors. A synthesis of issues researchers should be aware of: Test Content. Examinee Readiness Motivation and Response Set. Standardization Level. Examiner Bias and Language Differences. Reliability and Validity Issues. Societal Inequities. Indigenous: Avoidance of Bias The Indigenous viewpoints about data collection reflect concerns about the imposition of the colonial worldview on the strategies that are used. Indigenous researchers are more inclined to see "data sources" as coresearchers. Indigenous research typically involves a more equalitarian relationship that is built on mutual respect. The Indigenous data collection strategies are influenced by the values of importance in the community: co-production of knowledge visibility for the people who offer their "data " reciprocity for sharing information and adherence to appropriate protocols..

[Audio] 12-8 Examine data collection in research using the questions provided. Questions for Critically Analyzing Data Collection Methodological Validity Do the items on the measurement instrument appear relevant to the life experiences of persons in a particular cultural context? Do the measurement items or tools have content relevance? Have the measures selected been validated against external criteria that are themselves culturally relevant? Are the constructs that are used developed within an appropriate cultural context? Have threats to generalization of causal connections been considered in terms of connections across persons and settings nonidentical treatments and other measures of effects? What evidence is provided of the quality of the data collection instruments in terms of the following? Reliability or dependability. Validity or credibility. Objectivity or confirmability. Freedom from bias based on gender race and ethnicity or disability. Are the procedures used by the test developers to establish reliability validity objectivity and fairness appropriate for the intended use of the proposed data collection techniques? Is the proposed data collection tool appropriate for the people and conditions of the proposed research? Given the research questions of the proposed research when and from whom is it best to collect information? Does the instrument contain language that is biased based on gender race and ethnicity class or disability? If observers are used what are the observers' qualifications? Were instruments explored for gender bias? In terms of race and ethnicity biases were the data collection instruments screened so that the test content reflected various racial and ethnic minority groups? If the instrument is to be or was used with people with disabilities was the accommodation made on the basis of a specific disability? Did the researcher consider their own prejudices and biases that might affect data collection? Were the various different cultural groups involved in planning implementing and reviewing the data collection instruments? In the results? Were multicultural issues addressed openly at all stages of the research process? In observational research was it possible or reasonable to use multiple observers or teams diverse in age gender or ethnicity?.