I think that you make some good points. But I take issue with your third point. People lie about things to researchers (or simply don’t know have some sort of self-knowledge) all the time. This is the whole concept of “revealed preference” in economics. Someone can say that they care about sweatshop labor, but do they actually make any effort to avoid buying products produced in sweatshops?
Not questioning the experiment subjects’ stated sexual identity just neuters the whole point of the study: is homophobia driven by repressed homosexual desire. If it is repressed, we should expect subjects to say they are straight even if they aren’t. Could the methodology be flawed? Sure! But there is nothing wrong with trying to actually measure the homosexual attraction of someone who says they are not so attracted.
Agree with your overall point, but a “revealed preference” isn’t necessarily a lie or lake of self-knowledge. A recovering alcoholic might have a revealed preference for alcohol but that doesn’t mean they’re lying when they say they don’t want it or that they’re unaware of the temptation they have for it (insane as this may sound, people have actually made this argument before). The whole economic concept rests on massive philosophical and psychological cans of worms about what defines a person’s identity and wants, which economists are happy to oversimplify and ignore. The average person can’t really be expected to track entire supply chains for every purchase they ever make, which is why we have regulations. Instead of having every individual track every part of the production of every purchase, we (as a society) assign someone the job of investigating the production process to see if there’s anything that we would find objectionable.
If a lot of people say that they have a problem with sweatshops, but then purchase goods made in sweatshops, you could argue that their behavior “reveals” their true preference, but it would be equally valid to say that what what they actually consciously express is their true preference and their failure to live up to it is driven by ignorance, succumbing to temptation, or regulatory failure.
That’s fair, but I get the feeling that the researchers came up with their conclusion before performing their study, and then interpreted their findings to fit that pre-supposed conclusion. The only thing this study can fairly claim is that some homophobic men may harbor homosexual desires. They’ve failed to demonstrate any causal linkage between those two attributes, but they’re heavily suggesting one exists. Maybe their abstract grossly oversimplifies things, but it seems to extrapolate their findings far beyond any reasonable conclusion in my opinion, and that makes me question their methods and motives more than I normally would. The publication date also raises flags, as the common pervasive sentiment about homosexuality was very different in 1996 than it is today. All of those things combined indicate to me that this study should be carefully considered with plenty of grains of salt at hand.
But to get back on topic a little bit - my original intent was to refute the notion that if someone has a problem with the methodology of a scientific study, then they must perform their own study and present evidence to support a contrary claim. The examples I listed are things it would be reasonable to expect a layman with solid critical thinking skills to point out as potential flaws in this particular study, potential areas to look further into, to confirm whether or not the study is scientifically sound.
I definitely agree with you on all counts there. A single underpowered study does not sound science make, even disregarding the authors’ potential biases.
I think that you make some good points. But I take issue with your third point. People lie about things to researchers (or simply don’t know have some sort of self-knowledge) all the time. This is the whole concept of “revealed preference” in economics. Someone can say that they care about sweatshop labor, but do they actually make any effort to avoid buying products produced in sweatshops?
Not questioning the experiment subjects’ stated sexual identity just neuters the whole point of the study: is homophobia driven by repressed homosexual desire. If it is repressed, we should expect subjects to say they are straight even if they aren’t. Could the methodology be flawed? Sure! But there is nothing wrong with trying to actually measure the homosexual attraction of someone who says they are not so attracted.
Agree with your overall point, but a “revealed preference” isn’t necessarily a lie or lake of self-knowledge. A recovering alcoholic might have a revealed preference for alcohol but that doesn’t mean they’re lying when they say they don’t want it or that they’re unaware of the temptation they have for it (insane as this may sound, people have actually made this argument before). The whole economic concept rests on massive philosophical and psychological cans of worms about what defines a person’s identity and wants, which economists are happy to oversimplify and ignore. The average person can’t really be expected to track entire supply chains for every purchase they ever make, which is why we have regulations. Instead of having every individual track every part of the production of every purchase, we (as a society) assign someone the job of investigating the production process to see if there’s anything that we would find objectionable.
If a lot of people say that they have a problem with sweatshops, but then purchase goods made in sweatshops, you could argue that their behavior “reveals” their true preference, but it would be equally valid to say that what what they actually consciously express is their true preference and their failure to live up to it is driven by ignorance, succumbing to temptation, or regulatory failure.
That’s fair, but I get the feeling that the researchers came up with their conclusion before performing their study, and then interpreted their findings to fit that pre-supposed conclusion. The only thing this study can fairly claim is that some homophobic men may harbor homosexual desires. They’ve failed to demonstrate any causal linkage between those two attributes, but they’re heavily suggesting one exists. Maybe their abstract grossly oversimplifies things, but it seems to extrapolate their findings far beyond any reasonable conclusion in my opinion, and that makes me question their methods and motives more than I normally would. The publication date also raises flags, as the common pervasive sentiment about homosexuality was very different in 1996 than it is today. All of those things combined indicate to me that this study should be carefully considered with plenty of grains of salt at hand.
But to get back on topic a little bit - my original intent was to refute the notion that if someone has a problem with the methodology of a scientific study, then they must perform their own study and present evidence to support a contrary claim. The examples I listed are things it would be reasonable to expect a layman with solid critical thinking skills to point out as potential flaws in this particular study, potential areas to look further into, to confirm whether or not the study is scientifically sound.
I definitely agree with you on all counts there. A single underpowered study does not sound science make, even disregarding the authors’ potential biases.