{"id":4291,"date":"2018-09-18T18:53:49","date_gmt":"2018-09-18T22:53:49","guid":{"rendered":"https:\/\/www.amyork.ca\/academic\/zz\/?p=4291"},"modified":"2018-09-18T18:57:48","modified_gmt":"2018-09-18T22:57:48","slug":"test-bias","status":"publish","type":"post","link":"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/test-bias\/","title":{"rendered":"Test Bias"},"content":{"rendered":"<ul>\n<li>Biases that systematically <u>obscure<\/u> or <u>cause <\/u>differences among groups of respondent test scores <u>unjustifiably <\/u><\/li>\n<li>It is important that test scores <u>do not discriminate unjustifiability<\/u> against any particular group<\/li>\n<li>In this context, almost invariably \u2018group\u2019 refers to gender and\/or race<\/li>\n<\/ul>\n<p><strong>Gender differences in mathematics:<\/strong><\/p>\n<ul>\n<li>Females on average tend to score lower on mathematics in comparison to males<\/li>\n<li>However is it possible that conventional mathematics tests discriminate against females? \uf0b7 If so, then the differences should not be interpreted as a construct level difference<\/li>\n<\/ul>\n<p><strong>Construct bias:<\/strong><\/p>\n<ul>\n<li>Concerns the relationship of observed scores to true scores on a psychological test o Different factorial validity between groups<\/li>\n<li>If the relationship is systemically different for different groups, then we might conclude that the test is biased o In some cases, two people with different observed scores may actually have the <u>same level of the construct<\/u><\/li>\n<\/ul>\n<p>o Even when the two means are equal, construct bias still must be tested (bias may result in the equal mean)<\/p>\n<p><strong>Predictive bias:<\/strong><\/p>\n<ul>\n<li>rccurs when a test\u2019s use has different implications for two or more groups<\/li>\n<li>g. using ATAR to predictive first year university performance o If it were discovered that the ATAR was differentially predictive of academic performance for males in comparison to females, than that would be an example of predictive bias<\/li>\n<li>Construct bias does not necessarily imply predictive bias<\/li>\n<\/ul>\n<p><strong>Identifying test score bias:<\/strong><\/p>\n<ul>\n<li>There are at least two categories that can be used to identify test score bias o Internal methods to identify construct bias o External methods to identify predictive bias<\/li>\n<li>No single methods can be used to establish bias<\/li>\n<li>This is analogous to establishing validity for test scores<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>Test bias is about interpreting scores (validity) and are those results justifiable across all groups<\/p>\n<p><strong>Test score bias vs. true group differences:<\/strong><\/p>\n<ul>\n<li>Simply because two groups differ in their mean scores on a test does not imply that the test suffers from test score bias<\/li>\n<li>If women score higher on a scale that putatively measures openness to feelings, then it is fully possible that they are as a group, more open to feelings than men<\/li>\n<li>Similarly, a weighing scale would be expected to demonstrate that on average, males are heavier than females<\/li>\n<li>This does not mean that the scale is biased<\/li>\n<\/ul>\n<p><strong>Detecting construct bias- internal evaluation:<\/strong><\/p>\n<ul>\n<li>Construct bias only occurs through internal evaluation<\/li>\n<li>Evaluating construct bias typically occurs by conducting analyse at the item level \uf0b7 An item on a test is biased if:<\/li>\n<\/ul>\n<p>o Persons belonging to different groups responded in different ways to the item and o It could be shown that the differing responses were not related to the group differences associated with the psychological attribute putatively measured by the test<\/p>\n<p>\uf0a7 Variance not related to the true construct<\/p>\n<ul>\n<li>In plain language, a test is considered biased if you take two people (a male and female) who have the same level of an attributed (e.g. openness to emotions) but tend to respond differently to that particular item o Construct bias is present when different groups perceive the item differently for some reason<\/li>\n<\/ul>\n<p><strong>Approach to evaluating item bias:<\/strong><\/p>\n<ul>\n<li>The procedure focuses on the internal structure of the test o No need to collect data from other measures\/occasions<\/li>\n<li>Internal structure refers to the o Pattern of correlations among items and\/or<\/li>\n<\/ul>\n<p>o The correlations between each item and the total test score<\/p>\n<ul>\n<li>If the two groups exhibit different internal structures to their test responses, then we can conclude that the test is likely to suffer from construct bias<\/li>\n<\/ul>\n<p><strong>Factor analysis and bias:<\/strong><\/p>\n<ul>\n<li>Also known as PCA, is a data technique used to o Determine the number of dimensions associated with a collection of items (or subscales)<\/li>\n<\/ul>\n<p>o Define the nature of the dimension based on the factor loadings Construct bias is present if:<\/p>\n<p>A different number of dimensions is associated with test scores when a factor analysis is performance on the groups separately<\/p>\n<p>\uf0a7 e.g. when\u00a0 measured separately, one dimension found for men and two dimensions found for women<\/p>\n<ul>\n<li>more subtle construct bias can be present even when the groups are associated with the same number of dimensions (happens more often, same number of dimensions but the factor loadings present the bias; specifically the pattern of factor loadings is different between the groups)<\/li>\n<li>we estimate this with a factor congruence coefficient\n<ul>\n<li>coefficient should be positive and large to indicate the absence of construct bias<\/li>\n<li>measure this by correlating the component loadings between two items for both male and females<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>Factor congruence coefficient:<\/strong><\/p>\n<ul>\n<li>There are no published guidelines to help interpret factor congruence coefficients \uf0b7 The correlation however should at least be positive<\/li>\n<\/ul>\n<p><strong>Differential item functioning (DIF) analysis:<\/strong><\/p>\n<ul>\n<li>DIF is a sophisticated approach to the assessment of construct bias developed within the context of item response theory (IRT)<\/li>\n<li>Theoretically DIF assumes that we can estimate a person\u2019s trait level on a particular dimension via the test data<\/li>\n<li>This is true for all groups<\/li>\n<li>We wish to determine whether the trait levels and the item responses match up in the same way for both groups<\/li>\n<li>If they don\u2019t we have evidence for construct bias<\/li>\n<\/ul>\n<p><strong>Rank order consistency:<\/strong><\/p>\n<ul>\n<li>A simple way to test for evidence for construct bias o Calculate the means associated with each item separately for each group o Then calculate Spearman\u2019s R correlation between the item means (i.e. item difficulties)<\/li>\n<li>If the correlation between item means is low, then we would have some evidence to suggest construct bias<\/li>\n<\/ul>\n<p>Suggests that a low correlation between item means would be less than .90, results in construct bias<\/p>\n<p><strong>Detecting predictive bias- external evaluation:<\/strong><\/p>\n<ul>\n<li>Predictive bias only occurs in external evaluation<\/li>\n<li>Construct bias tends to be more in pure research contexts, predictive bias by contrast is a more applied consideration<\/li>\n<li>Specifically, do the scores from a particular test predict a criterion (or outcome) with equal accuracy for two or more groups<\/li>\n<li>g. government employees who creates the tests behind the ATAR should make sure that the scores have equal predictive validity with respect to university grade performance across all groups<\/li>\n<\/ul>\n<p><strong>Predictive bias analyses:<\/strong><\/p>\n<ul>\n<li>Involves two steps:\n<ul>\n<li>Need to determine whether the test scores predict the dependent variable to start with<\/li>\n<li>Need to determine whether the test scores predict the dependent variable equally across groups<\/li>\n<\/ul>\n<\/li>\n<li>Both steps involve conducting a regression analysis<\/li>\n<\/ul>\n<p><strong>Bivariate regression review:<\/strong><\/p>\n<ul>\n<li>Is an extension of Pearson correlation<\/li>\n<li>Consists of using one continuously measured variable to predict another continuously measured variable<\/li>\n<li>The main difference is that bivariate regression includes additional terms which can be used to create a regression equation \uf0b7 Intercept:\n<ul>\n<li>The expected value of Y when X=0<\/li>\n<li>The point at which the regression line crosses the Y-axis<\/li>\n<\/ul>\n<\/li>\n<li>Slope:<\/li>\n<\/ul>\n<p>o The expected increase or decrease in the dependent variable (Y) as a function of an increase or decrease in the independent variable (X)<\/p>\n<ul>\n<li>Predicting people\u2019s scores o Intercept is the minimum achievable score o Slope is the increase in dependant score after increase in independent score<\/li>\n<\/ul>\n<p><strong>One size fits all:<\/strong><\/p>\n<ul>\n<li>The assumption that different groups share a common regression equation is based on the idea that \u2018one size fits all\u2019<\/li>\n<li>When this is observed, there is no predictive bias associated with the test scores<\/li>\n<li>The regression equation derived from the total sample is referred to as the \u2018common regression equation\u2019<\/li>\n<li>Also use the terms \u2018slope intercept\u2019 and \u2018common slope\u2019<\/li>\n<\/ul>\n<p>Next we estimate separate regression equations for each group<\/p>\n<p>&nbsp;<\/p>\n<p>We then compare the group=level regression equations with the common regression equation<\/p>\n<ul>\n<li>If the group-level estimates do not match the common regression equation, then you might suspect that your test scores are biased<\/li>\n<\/ul>\n<p><strong>Testing predictive bias statistically- intercept bias:<\/strong><\/p>\n<ul>\n<li>If the slop is equal for both groups, can confirm there is no bias \uf0b7 What if the intercepts are different?<\/li>\n<\/ul>\n<p>o This suggests intercept bias e.g. if the male intercept is higher, suggests that males are rated higher than females<\/p>\n<ul>\n<li>If the lines run parallel, no indication of slope bias<\/li>\n<\/ul>\n<p><strong>Slope bias:<\/strong><\/p>\n<ul>\n<li>Suppose two groups had similar intercepts, but their slops differed in magnitude<\/li>\n<li>This implies that we would expect different predicted scores for males and females with the same test score<\/li>\n<\/ul>\n<p><strong>Intercept and slope bias:<\/strong><\/p>\n<ul>\n<li>There are occasions where both intercept and slop bias co-occur<\/li>\n<li>Rare to have only one slope\/intercept bias but not the other<\/li>\n<\/ul>\n<p><strong>Outcome score bias:<\/strong><\/p>\n<ul>\n<li>Thus far, the discussion of bias has focused on the predictor variable (e.g. test\/questionnaire)<\/li>\n<li>It is fully possible that the outcome variable scores are the ones that are biased (e.g. level of relationships, testers mood, supervisor ratings)<\/li>\n<li>For a full analysis of bias, you would want to examine both possibilities- predictor bias and outcome bias<\/li>\n<\/ul>\n<p><strong>The effect of reliability:<\/strong><\/p>\n<ul>\n<li>Are occasions where one group will have less reliable scores than the other group (unlikely to ever be exactly equal)<\/li>\n<li>The differences in reliabilities can cause differences in slopes and intercepts<\/li>\n<li>Differences in reliability are a form of construct bias than can have an impact on predictive bias<\/li>\n<\/ul>\n<p><strong>Keep in mind:<\/strong><\/p>\n<ul>\n<li>The absence of the difference between two means might be caused by construct bias<\/li>\n<li>We should probably always test the possibility of construct bias when we test the difference between two means based on composite scores<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Biases that systematically obscure or cause differences among groups of respondent test scores unjustifiably It is important that test scores do not discriminate unjustifiability against any particular group In&hellip; <a class=\"continue\" href=\"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/test-bias\/\">Continue Reading<span> Test Bias<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[107],"tags":[],"_links":{"self":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4291"}],"collection":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/comments?post=4291"}],"version-history":[{"count":0,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4291\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/media?parent=4291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/categories?post=4291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/tags?post=4291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}