in partnership with South Asia Center
(English captions & Hindi subtitles available)
About the Seminar:
Professor Jayachandran (in collaboration with Diva Dhar and Tarun Jain) evaluates an intervention in India that engaged high school students in classroom discussions aimed at eroding their support for restrictive gender norms. Using a randomized controlled trial, she finds that the program made attitudes 0.18 standard deviations more supportive of gender equality. The effects observed in the short run were still present two years after the program had ended.
About the Speaker:
Seema Jayachandran is a Professor of Economics at Northwestern University. Her research focuses on economic issues in developing countries, including environmental conservation, gender equality, labor markets, health, and education. She is a recipient of a Sloan Research Fellowship, National Science Foundation Early Career Development Award, and the Ecological Society of America's Sustainability Science Award. Professor Jayachandran currently serves on the board of directors for the Abdul Latif Jameel Poverty Action Lab (J-PAL) and is the chair of J-PAL's gender sector. She is co-editor for the American Economic Journal: Applied Economics and co-director of the National Bureau of Economic Research's program in Development Economics. In addition, she writes regularly for The New York Times as a contributor to the Economic View column.
Prior to joining Northwestern, she was a faculty member at Stanford University. She earned a Ph.D in economics from Harvard University, a master’s degree in physics and philosophy from the University of Oxford (where she was a Marshall Scholar), and a bachelor's degree in electrical engineering from MIT.
FULL TRANSCRIPT:
Tariq Thachil:
Okay. Hi, everyone. Welcome to CASI, the Center for Advanced Study of India, at the University of Pennsylvania, and to our weekly seminar series. I'm Tariq Thachil. I'm the Director of CASI. And thank you for joining us again for another edition of our Thursday seminars.
We are delighted this week to have Seema Jayachandran with us. And she is the Breen Family Professor of Economics at Northwestern University. Her research focuses on economic issues in developing countries, including environmental conservation, gender equality, labor markets, health, and education. She's a recipient of the Sloan Research Fellowship, a National Science Foundation Early Career Development Award, and Ecological Society of America Sustainability Science Award.
Seema currently serves on the board of directors for J-PAL, and is the chair of J-PAL's gender sector. That's the Abdul Latif Jameel Poverty Action Lab. And she's also co-editor of the American Economic Journal's Applied Economics, and a co-director of the National Bureau of Economic Researchers' program in development economics. She's published very widely on too many topics for me to outline here. But some of her recent research that specifically focuses on India includes a 2017 paper on child stunting entitled Why Are Indian Children so Short? The Role of Birth Order and Son Preference, which she co-authored with Rohini Pande who many of you delivered our Khemka Distinguished Lecture back in the fall.
She has a paper examining intergenerational transmission of discretionary gender attitudes with Diva Dhar and Tarun Jain in the Journal of Development Studies. That is related closely to what she's going to be presenting today. And then a recent working paper that was quite interesting co-authored with Monica Biradavolu and Jan Cooper that combines qualitative research techniques with machine learning to construct a survey module and index for measuring female agency.
So we're delighted that she'd come and talk to us today. It's also impressive that she writes in addition to for academic audiences, regularly for a general audience in her role as a contributing writer for the New York Times. So Seema, we're delighted to have you join us here at CASI. Just before we get started, most of you know the rules, but Seema's going to present for the first half of our time together. And then after that, we'll do Q&A.
If you have questions, I'd really prefer if you could just type them out and propose them to me directly in a direct message on chat, Tariq Thachil. And then I'll keep a list of questions. And I'll call on you to ask your question directly to Seema. And we'll try and get to as many of them as possibly, so do keep your questions as brief as you can. Seema, thanks again so much for joining us. Go ahead.
Seema Jayachandran:
All right. Thanks. Thanks so much Tariq. Thanks everybody for coming and for the invitation to present. Let's see. I'm on the wrong slide. This is work with Diva Dhar and Tarun Jain. And it's the study in the Journal of Development Studies was using the baseline data from this project, which is a randomized control trial. And so today I'm going to present the results of this intervention. And the goal of the intervention was to try to change adolescents' gender attitudes to support more gender equality through schools.
Okay. So just as some big picture Motivation, which given this audience maybe is not necessary, but there's a great deal of gender inequality in India, particularly in certain states in North India. So this project's in Haryana where the sex ratio at birth is about 1.2 boys for every girl. And female labor force participation is less than 25%. And it's been declining in recent decades.
So both the sex ratio and female labor force participation are aspects of gender equality that most governments seem to want to address, either because on league tables they come out on the bottom, or I think just having millions of young men who would like to be married and are struggling to find a wife, that's not good for civil rest in a society. And in terms of female labor force participation, India's done quite well in closing gender gaps in secondary school. And so one of the fastest ways to increase GDP per capita would probably be to deploy some of that human capital into the labor market.
This project started with kind of outreach between the government of Haryana and J-PAL South Asia about various challenges it was facing where it was interested in doing research. The government was particularly interesting in the sex ratio. They're doing many things already. It's banned. It's a law that has some serious enforcement behind it. And there are policies that give financial incentives to have daughters. So if a family has daughters and the woman gets sterilized or either of the couple gets sterilized, then you get some payments, often in the daughter's name.
So the specific way this started was the government was interested in changing and evaluating those financial incentives. And I think this is the first project I started after I had tenure, and I think that's reflected in several ways. One is it's a very long-run project with a long horizon. And it was also kind of an opportunity to say, "Okay, I'm going to do something that I'm personally keen on testing." So we counter-proposed to the government, rather than evaluating those financial incentives, will you allow us to try some other approach? Can we just try something new that you're not doing and evaluate it as a pilot? Which was to try to change the attitudes.
Again, the genesis was around sex selective abortion, but the project very much broadened. I don't think sex selection as really the main aspect of gender equality will end up changing. But the idea we proposed was to use the school curriculum to change gender attitudes. So in this project, an NGO designed the program and ran the program. It was done in government schools. And the government gave permission to the NGO to run these classes during the regular school day.
I'll say more about the intervention, but it was really about kids having a discussion. And the hypothesis was, if kids are just given time to think about some of the gender gaps in their society, why they came about, why they might be problematic, that thinking about and discussing them will shift their attitudes to be more supportive of gender equality. And some of the messaging is about the human rights aspect of equality. Some are pragmatic or economic arguments for equality. So it's a guided discussion. It's not a lecture base, but it's also not neutral in that the facilitator or teacher certainly has a point of view and kind of goal of shifting people in one direction and not the other.
So the intervention was focused on both boys and girls who were in grades seven to ten. So it was a two-year program, two and a half years. One cohort joined when they were in grade seven, and they were in the program in grade seven, eight, and part of grade nine. And the cohort above them was also in the program. And the kind of dosage was about one 45-minute to an hour session every three weeks over the school year, so ultimately 27 sessions, a little bit less than 27 hours over this time period.
I'm going to show you results from immediately after the program ended and then two years after it ended, so sort of five years from our baseline to our second end line. But the goal is to keep tracking the sample as long as we're able to find them and we're able to raise money. Because I think the ultimate goal is to shape the decisions they make as young adults like college completion for the female participants, age of marriage, whether the women who are in the program are employed, but also the wives of the husbands of the men who are in the program, and then the sex ratio of their children.
So why do we do this with adolescents and in schools? I think the idea of changing attitudes was the starting point. And then thinking about how to change attitudes, there's a view in psychology and maybe conventional wisdom that younger people have more malleable views. And in this case, at least for this format of an intervention, it was valuable that they were mature enough to have debates in class about these issues.
And then in terms of how are you going to reach adolescents, schools are a really powerful way for a government to provide messaging, in this case some kind of moral civil rights messaging, to the next generation. That in this case might be counterbalancing some messages that they're getting at home or in the community. So this is a case where the government was ahead of a lot of its adult citizens on this human rights issue. So the schools are pretty useful in that if they want to reshape those views.
So we did this with an NGO delivering it, and mostly as a proof of concept, I think, without the idea that that was going to be the model to scale it up. I'll talk about this at the very end or in the Q&A, but I think it could be potentially cost-effective to scale up if it was taught by regular teachers or some of the curricular materials were embedded in textbooks or audio/video material.
So the intervention was designed and implemented by a gender human rights NGO called Breakthrough. And as I mentioned, it was 27 sessions where the Breakthrough facilitators led the discussions and activities. So the teacher, regular government teacher, this was like during the school day, the teacher could choose to help out and participate. They could take a break. There was a mix of what they did.
There are 150 intervention schools, and each of the breakthrough facilitators covered 10 schools. So they would kind of rotate, spend one day at one school, then another school, and rotate, and sort of on this three-week cycle. Of the 10 schools per facilitator, there were 15 facilitators overall. And one thing to note is 13 of them were male. That's kind of part and parcel of the problem. This is a job where you had to work across a reasonably large geographic area, and so it was easier to recruit men than women for that role. But we don't have statistical power to see. I think it's an interesting question, what's the best gender to teach this class. We don't really have that much variation.
There were various topics. Some were about kids sharing their attitudes and discussing why they believed certain things were right or wrong, discussing their own personal aspirations, and seeing how if there were any patterns by gender. They talked about division of work, not just your own behavior but tolerance and discrimination of others. And then there were a few sessions on communication skills. Kind of under the view that, yeah, sure, you can change the person's attitudes, but to actually translate that into behavior, they probably need to persuade other people to convince their parents they should be able to go to college. And so there were some sort of communication skills with that in mind. And that theme of when are attitudes sufficient to change behavior is one I'll come back to a couple times during the talk.
This is a picture from Breakthrough, so I'm sure cherry picked to have more participation than usual. But it is indicative of the way the sessions were run. And I think part of why they worked is probably because most students seemed to enjoy them because it was more of an opportunity to share their own opinions and debate things than was typical in their high school day where it tends to be more being talked at. So there are a bunch of sessions, but one that I think resonates with me the most is one on household chores, because it sort of indicates or demonstrates how even without being too directed, maybe some kids will have an ah-ha moment and change how they think about gender equality.
So there's a session on chores where they break out into groups and list who does various chores in their household, like who does the cooking and the cleaning and the laundry. And then they come back together and share their answers. Lo and behold, it's always the women and the girls who do all of those chores in the household. And so then the facilitator says, "Okay. Why is that?" And then usually someone will say, "Well, women are just better at cooking." And then the facilitator will say, "Well, if they're better at cooking, who's doing the cooking in restaurants?" And the answer is men. So why is that? If women are really talented at this, why are we choosing to have men do it exclusively in restaurants? Do women get as much respect if they're doing it at home as in the restaurants? And you can sort of make a point about efficiency, that if you have these talents and you could feed hundreds of people at weddings instead of just your home, just your family, more people could enjoy those talents. And you can also see how that might bring you joy to spread your talents and share your talents with more people.
So not everybody's going to have an ah-ha moment. But I think for many kids, they probably haven't interrogated these views in much detail, and they haven't thought about it, or kind of had some view that when they're forced to think about it a little bit more they realize there's some inconsistency in it. You can imagine some students having a backlash and having less support for gender equality. But for me, I could see how this could be eye opening for some kids. So that's the idea, that they'll be thinking about it and noticing some things that they think are wrong, and therefore changing their attitudes.
Okay. So there's kind of a bundle of what this intervention is doing. The study wasn't designed to really isolate the channels. So I think first and foremost, it's trying to change people's preferences. In economics, we usually think of preferences as sort of given, but there's also research on different policies that shape preferences. And I think first and foremost, this was trying to make a boy, when he grows up, to have less disutility from his wife working.
It also changed people's sort of factual knowledge. If someone really hadn't thought about the money you're leaving on the table by not allowing women's talents to be used, then that's less about kind of a preference and more just belief or knowledge. And then there were the skills I mentioned around communication.
This is a randomized control trial. And the intervention is at the school level. We worked in quite a few of the government secondary schools. We excluded the very small ones. And we sampled at most one school per villages. So that left us w 314 government secondary schools across these four districts in Haryana. So this is New Delhi, so these are districts close to New Delhi. 215 of them were co-ed, and 99 were single sex, some all girls, some all boys.
And preview, one thing you might wonder when I show you the results, we don't find significant differences in the program effects across co-ed versus single sex. Though I can imagine reasons why it might work better in co-ed or it might work better in single sex. Breakthrough had money for 150 treatment schools, so in our randomization we chose 150 treatment schools, and then 164 are going to be the control group.
So this is the timeline. We started in 2013 with a baseline survey. The school year starts in April, so the intervention ran from April 2014 to September of 2016. And then we did our first endline survey in about a few months, maybe on average three, four months after the intervention ended in 2016-2017. And then we did another one two years later where there hadn't been interaction between the NGO or us and the participants over that time period.
Because we were interested in long-term outcomes, we have a big sample. Like it's kind of ... What I'm going to show you, we're overpowered for the immediate results. So we enrolled 14,809 students. And how did we choose those students? We gave consent forms to all the kids. And then we randomly chose from among those who consented to participate and whose parents consented.
One thing you might thing is parental consent might give us a skewed sample. For better or for worse, we don't see any patterns between the consent rate and things like the level of gender equality, or consent rate for girls versus boys. I think we got back over 80%, close to 90%, of the forms. And most of the times we didn't, the student said they lost it or forgot it. And kind of more broadly, parents could consent out of our study, but they couldn't consent out of the intervention. That was the government of Haryana's decision to add this to the schools. But we did a lot of monitoring, thinking parents would object or there'd just generally be backlash from the community, and we didn't see that. So I don't have an explanation for why that is, but certainly the sort of stylized story of the community and parents have these retrograde attitudes, and the government's going to come in and do this intervention, I expected more pushback. And we didn't see it, at least at the scale of this intervention. It doesn't mean it wouldn't arise if you scaled up.
Okay. So here are some characteristics of the students. So first of all, we have balance between the treatment and control group. And then the kids were 12 at baseline, so they're going to be 15 when I show you for the first endline, and they're going to be 17 years old on average at the second endline. This is a mostly Hindu part of India. And female labor force participation is below 30%, consistent with the rest of India.
So we pre-specified three main outcomes. They're each going to be an index of several survey questions. I think of the three primary outcomes, I think the one where we have the primary of the primary is probably the first one, which is the gender attitudes. So I think this is the one where we can measure it most comprehensively. And so we have attitudes about education and employment. And most of them are rural value surveys or other previously vetted questions.
There's a second outcome that's the only one that's specific to one gender where we only had a hypothesis it would change for girls, which is their aspirations toward education and career. And then the third one is self-reported behavior where we ask questions ... I think ultimately the goal of this intervention is to change behavior too. And I think where we're better placed to measure that is going to be when the kids are adults and they're making life decisions around marriage or working. And we can kind of ask people and we'll probably get honest answers of whether they're employed.
But right now, when they're 15 or 17, they're still living at home and don't have a ton of autonomy. So we focused on behaviors where we thought kids have some degree of choice in that behavior and that where they're governed by gender norms. So I would say this is probably the [inaudible 00:21:08] of the outcomes which is some questions about doing household chores, about interacting with the opposite gender, and about how they interact with their parents, regarding older sisters and sort of supporting their sisters' college ambitions or career ambitions.
We also pre-specified how we would analyze this. It's pretty simple. We're going to have this index as an outcome. We're going to have a dummy variable for being a treatment school. We'll control for the baseline analog of that outcome and some fixed effects.
Okay. So I can present the main results from this paper in one slide because it's kind of a simple RCT, and we pre-specified these three outcomes. These are all in standard deviations. The control group has a standard deviation of one. And everything's coded, so a higher value is the direction the intervention was aiming to move things in. So more support for gender equality, higher aspirations for girls, and behaviors that are more consistent with gender equal norms.
The bar on the left are attitudes. And we see a 0.18 standard deviation improvement in gender attitudes. We don't see any impact on girls' aspirations. Interestingly, girls' aspirations are almost as high as boys' aspirations. So in some sense, there wasn't that much room for improvement. I hesitate to kind of be normative and say girls' aspirations were too high, but they were certainly kind of higher than what ... In the control group, it's unlikely that, say, 85% of the women are going to be working after marriage, but their aspirations were that high. And then we also see 0.2 standard deviation increase in behavior aligned with gender equality.
So just to say a little bit more on effect sizes, because I know standard deviations are hard to wrap your head around. So the treatment effect, I'm going to focus on attitudes, the treatment effect of 0.18 standard deviations, one metric, or one benchmark, is that paper where we had at baseline for a sub-sample, we surveyed some parents. So at baseline, we have a measure of parents' attitudes and the kids' attitudes. And we can put in a bunch of fixed effects and get at least a pretty robust association between a parents' attitudes and a child's attitude.
So when a parent is one standard deviation more supportive of gender equality, the child is 0.11 standard deviations more supportive of gender equality. And so the treatment effect is larger than that. I don't want to make too much of this, but on the one hand, it's kind of surprising because they've been living with their parents for 12 years. But I do doubt they had spent 27 hours kind of explicitly thinking about it. So I think it's kind of interesting to think about the contrast, the sort of osmosis way of learning for parents versus this active learning in the classroom.
Another benchmark is the girl-boy gap in the control group in attitude. So girls, just take the control group, girls have more support for gender equality than boys, not surprisingly, and that gap is 0.5 standard deviations. So the treatment effect is not that big, but it's reasonably big relative to that girl-boy gap. And I think my favorite way to think about the effect size is as a persuasion rate. Which is thinking about all of the questions, of these 18 questions and all of the individuals, and thinking about the cases where there was room for improvement, so coding them as binary, how many times of the cases where someone was not supporting gender equality, what percentage of them were converted by the intervention? And the answer is 16%. So it doesn't solve all of the problem, but that's not surprising. This is not simple enough of a problem to be solved by one intervention. I also think even with this approach of using schools and the curriculum, one can imagine increasing the dosage. So in that sense, 16% seems even bigger in terms of the overall potential of this strategy to change attitudes.
Okay. I don't know how I'm doing on time. Okay. One thing that I'm sure people have been wondering about is, could this be just social desirability bias, experimenter demand, where the treatment group, they've been in an intervention that was ra-ra gender equality, and maybe they said they supported gender equality just to look good to the surveyor. One aspect of what you say, changing what you say, even if it's not what you change in your heart of heart, I think is valuable. If people keep to themselves sexist views or racist views, that has value. But I think what we'd be worried about is if they didn't do that in their real life and they were just saying this to our surveyors because they could deduce that this was related to this intervention they were in.
So what my colleague at Northwestern Alice Eagly recommended to me this tool that social psychologists use of the Crowne-Marlowe survey instrument to measure an individual's propensity to give socially desirable answers. And so we added that to our baseline survey, and we're going to use that for heterogeneity analysis. So what would be worrisome is if all of that treatment effect we saw was driven by the people with a very high propensity to give socially desirable answers. And luckily, we don't see that. We see that even the people who have a low propensity, we still see an equally strong treatment effect.
If you haven't seen this scale before, I love it. I think it's super clever. It's not asking anything related to gender equality. It's just asking you on a yes or no scale, so not on a Likert scale, we use a 13-question version of it, if you have these too good to be true traits. I've never been jealous of somebody else's good fortune. And so if you were honest, you're either saintly or your answer to most of these is that you're not having this perfect personality trait. And so most of the variation ... The assumption is the variation in this is often just kind of people trying to look good to the surveyor or trying to delude themselves.
So this is how we use it. I like it because it seems to work in the sense of this coefficient here, this is a binary for having a low score on that scale, so a low propensity to give socially desirable answers. And we see that those people report less support for gender equality. And so what is this telling us? That overall in the sample, there is some shading up of answers. People are stating they support gender equality partly to look good. But that's kind of overall for our sample.
What would have been worrisome about how to interpret these findings is if we had seen a big negative interaction effect here. That would've said that all of our effect is from people who have this high propensity to give socially desirable answers. And the effect goals away if we focus on the people who seem to be giving more honest answers. Instead, we're seeing basically a zero interaction effect. And we're seeing the same effect regardless of this. So I don't think it nails this question, but for me, I updated a lot based on this. The next thing I'm going to show you is our long-run results in a couple slides. And I think the fact that the results persist also kind of alleviates some of the concern that if you think the social desirability bias would fade over time.
In terms of heterogeneity, we see the same effect size on attitudes for boys and girls. Girls start out with more support for gender equality, but they move up. Both genders move up. Where we see significant differences across genders is on behavior.
Here I think my interpretation of this is that if you change your attitudes, you also need to have the freedom or power in society to act on those attitudes. And so this is a case where, unfortunately, boys have more of that power. So for better or for worse, this reinforces the point that it's important to involve boys or men in these programs that are trying to change gender norms. Because if you're able to change their attitudes, they might be able to act differently with more freedom than girls can.
Okay. So just briefly, we see persistence of the effects two years later. This is the same format as what I showed you before, and you could see the effect sizes are pretty similar. We see persistence on attitudes and behavior, and continue to see nothing on aspirations.
Here the gender heterogeneity is a little bit different. Now, even on attitudes, we see a bigger effect for boys than girls. So the attitude change persisted more for boys than girls. And you could do some ex post theorizing that maybe it's because the boys were acting on it and getting some positive reinforcement, whereas the girls were trying and failing, and therefore sort of faded back. I don't really know. We didn't anticipate this. We don't really have a great way to impact what's going on. I think it's interesting for further research to understand these dynamics over time that are different for boys and girls.
We also added two revealed preference measures in the second endline. The first was a petition opposing the dowry system. We sort of messed up on this in that we made it too hard to sign the petition. You had to make a phone call. So we were aiming for higher means than this. Treatment group has a slightly higher point estimate, but there's no difference between the treatment group and control group in terms of signing this petition.
Where we see a marginally significant effect is on a scholarship application for girls to go to college. So we created a scholarship program and had a tedious application. And girls who were in the treatment group were more likely to fill out that application. So that could reflect them wanting to go to college more intensely or expecting to be able to go to college because they had convinced their parents. And so even though we didn't see anything on our measures of aspirations, I think it could still be an aspirations effect here in that it's just a different, it's like a more granular measure. Where we asked, "Do you want to go to college," and everyone said yes both in the treatment group and control group, where you had to put in some costly effort to do this application, we start to see a difference.
Okay. So just to summarize, and then I'll have one slide on the next steps, we see this effect on the intervention making attitudes more supportive of gender equality for both boys and girls. We see bigger effects on behavior for boys, maybe because they face fewer societal constraints on how they can behave. We don't see any change in stated aspirations, though we do see this effect on this college application as a measure of aspirations or expectations. And then we see a persistence of the effect, including greater persistence for boys and girls. Though there is persistence for girls as well.
So in terms of next steps, one is we would love to continue following up the sample. Now our hope is to do a survey at the end of 2021 when the sample will be 20 years old. So that will be at the eight-year mark. And the completed college will be an outcome or at least college enrollment. The kids will be past the, about 50% to 60% of the girls will be married, and so you could see things like age of marriage.
I personally think that improvements in these hard outcomes like marriage age and college will be valuable for encouraging scale-up by governments. Sort of a scholarship application or an attitudes is a little bit fuzzy for a government to really want to pivot on something. But we're also interested in seeing if there are spillovers to the siblings or parents.
And then the other thing we've made some progress on in the last six months is scale-ups. J-PAL South Asian and Breakthrough have been in discussion with state governments. The chief secretary in the state of Punjab is interested in scaling it up through regular moral science and social studies teachers, where Breakthrough will train those teachers. And then we're in some earlier conversations with ODISHA, where they want to try a pilot and see how well this works with Breakthrough teaching it and then the government teacher shadowing it.
I mean, personally, the model I'd like to see is hiring a special purpose government teacher. So if you think like one extreme is the NGO, the other extreme is government teachers who weren't hired with this program in mind, an in between would be to hire government teachers who aren't specializing this like covering 20 schools each. And I guess the difference there is then they could be chosen to actually believe in gender equality. So one challenge with government teachers is they may or may not believe in this. And insofar as you want this discussion-based format, I think Breakthrough chose people who were good at leading discussions, which isn't necessarily how you choose moral science or social studies teachers.
This is one version of the program, and I think there are many other dimensions of the program that you can imagine varying to optimize. I don't think this was the final word on the optimal age or dosage for trying to change attitudes through schools. So that's it. Thank you very much.
Tariq Thachil:
Okay. Thanks so much, Seema, for that really interesting talk. We have a few questions. Can I just ask a quick clarifying question before we get to some of the questions here, just in terms of the intervention itself? You ask for this consent, and you get, whatever, 45 kids who consent per school. And then the kids who are selected for the intervention, when the intervention is administered, are they taken out of whatever classroom they're in and put in a different ... I'm just trying to think of the mechanics of how it was done. It was during school hours and they were taken out of the section they were in or the class that they were in? Or how did that actually work?
Seema Jayachandran:
Yeah. There's this distinction between the intervention and the study. In the schools, all of the kids in those grades, whether they consented to be in our study or not. And we had a lot more people consent, and then we randomized which ones were in our study. But everybody was in the intervention, so you couldn't opt out of that. That was a government program. And so nobody was taken out of the class. There was just not everybody who was in the program had to do our surveys.
Tariq Thachil:
Okay. All right. The first question we have is from Kimberly, and she says her internet connection is a little spotty, so I'm going to ask it on her behalf. I think she just was curious if you had maybe like a table or something that summarized what the different individual attitudes and behaviors were that you mentioning. I know you mentioned a few of them. But I think she was curious to know kind of more specifically. She says, "What specific attitudes of behaviors were you measuring?" So I don't know if you have that [crosstalk 00:37:13]-
Seema Jayachandran:
I don't have that in this slide deck. I can try and multitask and pull it up. It's an appendix table in the paper. If you go online, you could probably find it in [inaudible 00:37:23]. I probably have a slide. Rather than trying to multitask, let me just list the ones I remember. Some of them are a woman's most important role in life is to be a good homemaker. So that's a world value survey question. Or boys and girls should get equal opportunities for educational resources. Some of them were vignette based, and some of them were these direct questions. There were a few India-specific ones like, "I think it would be a good idea for my community to have a female sarpanch. A couple of them were around fertility, like whether families who have a only daughter, family that was wanting to have two kids and they only have daughters; should they stop or continue? And things like that. So sort of questions about not asking what they plan to do, but just what they think is right and wrong.
Tariq Thachil:
Okay. All right. Thank you. So the first question that somebody can ask directly is from [Hirscha 00:38:26]. Hirscha, do you want to ask a question?
Hirscha:
Sure. Hi, Seema. I have a question about ... I think it was early on when you presented the randomized trial design, it seemed like there were a substantial number of same sex schools. And I'm just curious, number one, just how do they intervention effects differ there, and also just qualitatively, do you have insights on how the curriculum and the delivery of it looks different when you have same sex classes? Are there level differences for the kids who were in those schools even to begin with?
Seema Jayachandran:
We don't see overall heterogeneity by whether the school is co-ed or not. And a caveat here is that the communities that have single sex schools, it's not random where there's single sex schools. So how you interpret that heterogeneity. But nonetheless, and I think when you see no heterogeneity, it could be because it's irrelevant. Or it could be because in some cases it works better in co-ed and in single sex.
And so certainly Breakthrough expected, like it works better in co-ed schools, because they think one of the big ah-ha moments for kids is girl realizing that girls also want to be, except for a cricket player, there's basically a lot of overlap in the career aspirations of boys and girls. So that's an ah-ha moment that you wouldn't have in a single sex school.
Hirscha:
Yeah. Exactly.
Seema Jayachandran:
On the other hand, you can imagine maybe more frank conversations if people are less shy when they're in a single sex environment. There's something interesting. That's heterogeneity across schools, but even a heterogeneity across kids. Just more broadly, I had a conversation with Breakthrough at the beginning of like, "Oh, wouldn't it be interesting to have one arm that focuses more on implicit messaging or explicit messaging, or the human rights arguments and the economic arguments?" And their practical answer, like response, is, "Well, some people are going to respond to one thing and another to another." So practically you would never do that. You kind of want to throw the kitchen sink because everybody learns differently. Anyway, that kind of zero heterogeneity, I think it's quite likely it's reflecting that for some boys, they're more honest if there are no girls around, and for other boys, it was eye opening to hear from their female classmates.
Hirscha:
Yeah. That makes sense. Thanks.
Tariq Thachil:
Okay. The next question is from Anahita. Anahita, do you want to ask your question?
Anahita:
Hi, Professor Jayachandran. Were there any predictors, or did you detect any predictors, of which children responded with a positive effect, such as household SES or mother's literacy, any preexisting predilection for any attitudes or gender quality in general?
Seema Jayachandran:
Yeah. So we pre-specified what we were going to look at for heterogeneity, so we haven't been exhaustive. But one thing I realized, when you do a pre-analysis plan, people still have lots of questions. So we've run a few things. I'll tell you what we've tried. What we pre-specified was gender, which I showed you, and then parents' attitude. So for a sub-sample, we survey the parents, and there we don't see any average effect of parents' attitudes. And that's another case where you could imagine it's more effective if you start out in a more conservative family, or it's more effective if you have a progressive family. But in any case, that wasn't an average predictor.
But we actually didn't see strong patterns across the other things we looked at, like whether you have sisters, or just the gender composition at home, or other measures of the communities' attitudes, like proxying it with, say, the sex ratio or the female employment rate from the census.
So one thing we could do, I guess, still consistent with [inaudible 00:42:16] to a pre-analysis plan is we could use machine learning techniques to see what comes out of our baseline characteristics as predictors of heterogeneity. I'm sure there is something. But the kind of usual suspects we looked at are audience members in early presentations most curious about, we didn't see individual predictors.
I should also add, we didn't ask a lot of psychological ones that I think are interesting. Like Alice Eagly the colleague I think had suggested, there's some sort of openness to new ideas or psychological concept that's like a different direction to go into in terms of who would be most receptive of this. We chose not to go down that route in where we focused our baseline survey.
Tariq Thachil:
Okay. The next question is from [Yajna 00:43:10]. Yajna, go ahead.
Yajna:
Hi, Professor Jayachandran. Thanks for your talk. The first question was kind of linked to what you already discussed earlier. I just wanted a bit more detail about the specific attitudes and behaviors that you tested. But related to that, I wondered if you specifically talked about inequities in the classroom itself, so sort of biases that students or teachers themselves might have, or sort of barriers that girls may face within school.
Seema Jayachandran:
Yeah. I'm trying to think if that was ... That wasn't a topic of a particular session. So what I know best is the kind of guidelines for the sessions that Breakthrough has written down for the teachers. I don't watch. And they didn't record, and I haven't seen videos of all the session. So it's quite possible that came up within the classroom.
There's certainly some discussion about harassment that happens at schools, like boys, whatever, pulling girls' hair. Like the unfortunate term of Eve teasing. So that is a topic, but that's not about teachers calling on boys more.
I think an interesting aspect of this ... I should say, we looked at school performance in aggregate because one of the education secretaries was worried that this is going to crowd out something and hurt learning. And I actually think it might help learning because this non-rote discussion might be valuable. But one prediction that people often ask is, did this help girls with their school performance? And I think a useful thing to remember is girls have higher board exam scores than boys, and they're studying more than boys.
So at least through the secondary school, it's not as if the lower opportunities to go to college is showing up in less effort or lower performance. That doesn't mean that there isn't ways in which girls feel constrained or discriminated against by the teachers or in the classroom. It's definitely not one where girls are on average looking worse in academic performance.
Tariq Thachil:
The next question is from Sumitra. Sumitra, go ahead.
Sumitra:
Thanks. I just had a couple of quick questions. So I was curious to know, is there anything happening with the control group at this time? Are they getting a placebo dosage of some sort, or is it just normal school hours for them? And whether any considerations like that went into the design. And then also, was there also consideration in the design about spillover effects between schools in nearby villages? Or are they too far [crosstalk 00:46:00] to be concerned about that?
Seema Jayachandran:
Yeah, both great questions. Let me start with the second one, which is, on the spillovers, that's why we chose, at most, one school per village, to try to minimize that. We actually haven't done analysis on trying to use GPS coordinates to see if there were spillovers, but that was partly based on anecdotally, it didn't seem like kids were having that much interaction with peers in neighboring villages. Especially since it had to be in the exact same plus or minus one grade or two grade, so their grade or one above or one below.
So we weren't too worried about spillovers as a source of contamination. We do have the GPS coordinates, and we could do more to look at that. There's another way of thinking about spillovers as like interesting to look at. And I think for our level of dosage, the most relevant type of spillover where we'd have power to look at is the one within families.
Now I've forgotten the first question. Just give me a brief reminder.
Tariq Thachil:
It's ... Oh, sorry, go ahead, Sumitra.
Sumitra:
I just want to know if there were design considerations [crosstalk 00:47:17]control group.
Seema Jayachandran:
Yeah, yeah, yeah. No, there was. We didn't have a placebo intervention. I think the study maybe think it'd be interesting just to see if you had any debate discussion class added. So if I had to have a placebo one, it might be a placebo conversation about environmentalism or some other topic. We didn't do that, partly because the government had to agree to add these sessions. And so if we said we want to add something that we don't have the resources to design, none of us are an expert at designing, but it's to have an apples and apples of discussion, one on gender, one not on gender, they probably would've pushed back. So yeah, our counterfactual here is just regular school. And it wasn't a set time, so it's some mixture of sometimes math, sometimes language or whatever that was getting crowded out because there was this 45-, 50-minute session on gender.
Tariq Thachil:
The next question is from Apulva. Apulva, go ahead.
Apulva:
Hi. I was just wondering if you could say a little more about the anti-dowry petition behavioral measure. I mean, I understand that you retrospectively now know that this is perhaps a little too hard to do, but surely then we should have expected the control to have done even less, right? The fact that similar numbers or proportions are calling to sign up to that petition should still ... I was just wondering if you had more to say about that particular measure. And I also didn't follow the numbers, so excuse me.
Seema Jayachandran:
Yeah, yeah, yeah. So what was the measure? Yeah, no, I agree with you that whether the level was 80% or 10% on average in the control group, we still have a hypothesis that there would've been a higher rate in the treatment group. So that hypothesis isn't confirmed by the data. The low level is only relevant for thinking about statistical power, that we would've had more statistical power to have seen a 20% increase in the treatment group if it had been around 50% rather than 80% or 10%. That's just about a dummy variable having a larger standard deviation.
Anyway, but what was the measure? I think some of these revealed preference measures that researchers create because some of them are still potentially subject to social desirability bias, even if they're a behavior. So we created a petition where we said we're going to send a list of the signatories to the local politicians, and we're going to run it in the newspaper. So we took out ads in the local newspapers and listed the names and villages of the students.
So our idea was like we want to do something that's more than just telling us you have to be willing to put this out in public. And so if it were just social desirability bias, you wouldn't do this. Well, we did it in person. Close to 100% of people said yes, they would sign it. So it's almost like someone's asking them, "Will you do this," and they did it. In our pilot anyway, I guess we were just worried that even though, yes, there is this real consequence, in the moment they just feel pressured to do it. So then we said we don't want to do it in the moment. We could've left it and have them come back, but that's expensive, so we had them make phone call. So you just try to guess, how many people have access to a phone call.
One interpretation is the drop-off from 90%, 100% saying yes in person to what we saw of 14% is that 85% is just disingenuous. I guess another is some of it was disingenuous, and some of it was like it was a pain. They lost a phone number, or it was a pain to get access to a phone or something like that.
Anyway, yes, the prediction would've been, if it was 14% in the control group, it should've been the same standard deviation effect sizes, like it should've been 17%, and we see 15%. But part of it is, at that point, you don't have much statistical power to see this 17%.
Tariq Thachil:
Okay. Thanks. We have a couple of questions still to go, so I think it may make sense to group a couple of questions right now and then have you answer them at once. The first is from Manya. Manya, if you're still there, do you want to ask your question?
Manya:
Yep. Thank you. I'm curious about [inaudible 00:52:23] about what led to the greater persistence of effects for the boys, and if there's variables like parental attitudes that seem to have been measured that also had a relationship with greater persistence in any way. And I also wanted to ask about facilitation, and to what extent, if any, there was variation based on which of the 15 facilitators were facilitating a lesson, and if there's any research on how effective Breakthrough's training is for government teachers. Thank you.
Seema Jayachandran:
Sure.
Tariq Thachil:
And then we have another question from Bhumi. Bhumi, do you want to ask your question?
Bhumi:
Sure. Thank you. Yeah. Related to this idea of why there may have been greater effects for boys, I was curious if you see any heterogeneous effects based on family dynamics. So I'm particularly wondering if children who live with nuclear versus extended families have different attitudes or behavioral outcomes, and if children who are firstborns versus those who aren't specifically have different behavioral outcomes.
Tariq Thachil:
Yeah. I think, in fact, just to piggyback on that, I think it was like 13 of the 15 facilitators were male, and so I was just curious about that and whether you thought that had any kind of impact on the discussion and who it was impacting, especially with these medium term effects. Go ahead, Seema.
Seema Jayachandran:
Sure. Yeah. In terms of the persistence, I don't think we can say anything too convincing in terms of data analysis to get at why there was more persistence for boys. We haven't looked at heterogeneity by the parental attitudes for that persistence, or tried to do heterogeneity analysis. And we can do that to see what are the predictors of where it persisted for boys. The one that people have speculated to us that I think makes sense is what I mentioned, which was like, there was this difference of if you believe in something and you try to act on it and you can't, it might be just harder to hold onto that, but that's just speculation.
And in terms of the male facilitators, I certainly have a prior that for boys they would be more influenced by a male, especially since these are sort of like young, cool men or good presenters, dynamic. So because we have 15, we can't really see what was the difference with those 13, those other two. But I think it's an interesting question for the research to see, do you need to have men? Especially if you're doing co-ed, you can't tailor. So what's the best way to do it? That might be related to why it stuck with the male participants more.
In terms of the 15 facilitators, we actually haven't estimated the facilitator fixed effects. I'm guessing there are significant facilitator fixed effects. They were also covering different geography. So just as a caveat, if we find facilitator fixed effects, we can't know if that's really geographic fixed effects, one district versus another.
And then in terms of Breakthrough's training, I think it's a question of how well ... I think where I've been pushing them is to try to codify their curriculum. I think this is kind of an interesting point about NGOs is that I think their instinct is to expand. And they've gotten a lot of funding to hire more people and do more of the program. But I think they'll have much more impact if instead they codify their program like create training videos, like have some excellent teachers tape versions of this so people can watch. The training is not just reading a manual, it's seeing things. And then other NGOs can pick that up, or it can help with government training.
So I think Breakthrough's very good at what they do. And so presumably, when they train, they'll train very well. But I also think, and this is not just unique to Breakthrough. Anyway, I think there's a lot of power for an NGO of saying, "I'm going to create the software," rather than just deliver the software and the hardware. There's a whole set of skills of how do you make that software better.
I think in the business training, Freedom from Hunger has done this. They did some business training. And now I think they just specialize in creating curriculum that other NGOs are using. And they probably impacted and helped more people's lives. And so I'm trying to convince Breakthrough that that should be their goal.
And in terms of the family dynamics, we looked at some, like whether you have an older sister. I think the hypothesis is that if you noticed things in your older sister hitting up against these walls, you might be more attuned to it, or these lessons are more alive for you. So we've looked at that. But we haven't looked at things, and we can, of like your grandparents live with you and aspects like or firstborn.
Tariq Thachil:
Okay. We've got two final questions. [Tanu 00:57:34]. Tanu [Shigoyu 00:57:34], go ahead.
Tanu:
Thanks, Tariq. In your talk, you shared many reflections about the scale. And that got me thinking if you thought about how you designed the curriculum and if you measured how much students retain, what elements were appealing to them. And then I had other question about the [inaudible 00:57:55] differences between girls and boys. And I think this point came up in several questions. And I wondered what the baseline attitudes were. Because I recall that you shared baseline attitudes about aspirations, but I think I missed learning about their baseline gender attitudes. And also my final question is, beyond attitudes, were there spillovers in the classroom in terms of how they interacted with their peers in co-ed schools or their friendship networks or choice for lab partners, so these kind of softer measures? I would be very curious to know that. Thank you.
Seema Jayachandran:
Yeah.
Tariq Thachil:
Okay. That's three questions. And I'm going to add a fourth. And you don't have to answer all of them. But just something that I've been thinking about is that you put a lot of stock, I think, in the presentation on the kind of debate and discussion that this curriculum and intervention provided as a way for kind of expanding critical thinking. But do you think it's just providing a different zone of authority for getting norms on these matters? That now by putting it into the school area, which is another zone of authority as opposed to parental authority? Because I think that does have some implications for scale up and whether you could deliver this through government schools, or whether [inaudible 00:59:02] oriented format. How much work are you confident that this particular style of curriculum is doing, versus just creating the school as a authority zone for creating these kinds of norms?
Seema Jayachandran:
Yeah. Great. I'll start with that question. I think that's right. That's my own projection of why do I think this was effective. Yeah. I think it's relevant just because I think we were worried especially boys would find this boring, like would disengage. And they didn't. So my best guess of why they didn't disengage was because of the format. But you might be right; like whatever is taught in school they'll engage with and pay attention to. So I think it's just a question we can't answer. But I agree with you that if you could move away from this style and just have it more like reading things or a lecture, then it becomes much easier to scale up, or you don't have to get as many things right to scale up. So that would be nice. I think that would be an interesting thing to test.
In terms of do people change their friendship networks, some of those variables we didn't collect all of those outcomes. Though I will say we did try one thing where in some of the classrooms, at the same time around the first endline, we created a little competition in some of the classrooms, and we were observing. We had two surveyors, and they ran the game. And then we looked at single sex versus co-ed group formation. And on that one, we saw no movement. It was just kind of incredible that when you ask kids to form, even after the treatment, to form groups of five to work with, 99% are single sex. It's just so extreme. So that one didn't change. Obviously, on things like who they study with or ask questions when they're stuck, we don't have measures of that.
In terms of the baseline attitudes, I'll say one thing, which might be kind of getting at what you're wondering about. Girls have more support for gender equality, and boys have less support, at baseline. And so we see the same treatment effect in standard deviations. But they're not starting from the same point. But in that result, we have controlled, or we've shown that it's similar when we control in parallel, when we also have an interaction of treatment times, your baseline attitudes. So basically adjusting for the fact that girls start out with more support for gender equality. We continue to see, if a girl and a boy start out at the same point, we still see the same patterns of improving the same amount or more persistence for boys in the second endline. So I think that's a case where standard deviations become problematic in interpreting that heterogeneity. But it doesn't seem to be what's explaining our gender patterns.
There's another question I've forgotten, but I also might be over time.
Tariq Thachil:
You're fine. I'm sure people can follow up with you if you want. But thanks so much for patiently sifting through all of our questions. And I think it's just a testament to the fact that you gave us a lot of interesting material to consider. Thanks for joining us virtually. And we hope we can have you back in the future in person. Thanks so much, Seema.
Seema Jayachandran:
Great. Thanks, Tariq. Thanks, everyone.
Tariq Thachil:
Bye.