White Corporate Feminism

When I first went to the Grace Hopper Celebration of Women in Computing conference, it was magical. Being a woman in tech means I’m often the only woman in a team of male engineers, and if there’s more than one woman on a team, it’s usually because we have a project manager or marketing person who is a woman.

Going to the Grace Hopper conference, and being surrounded by women engineers and women computer science students, allowed me to relax in a way that I could never do in a male-centric space. I could talk with other women who just understood things like the glass ceiling and having to be constantly on guard in order to “fit in” with male colleagues. I had crafted a persona, an armor of collared shirts and jeans, trained myself to interrupt in order to make my male colleagues listen, and lied to myself and others that I wasn’t interested in “girly” hobbies like sewing or knitting. At Grace Hopper, surrounded by women, I could stop pretending, and try to figure out how to just be myself. To take a breath, stop interrupting, and cherish the fact that I was listened to and given space to listen to others.

However, after a day or so, I began to feel uneasy about two particular aspects of the Grace Hopper conference. I felt uneasy watching how aggressively the corporate representatives at the booths tried to persuade the students to join their companies. You couldn’t walk into the ballroom for keynotes without going through a gauntlet of recruiters. When I looked around the ballroom at the faces of the women surrounding me, I realized the second thing that made me uneasy. Even though Grace Hopper was hosted in Atlanta that year, a city that is 56% African American, there weren’t that many women of color attending. We’ve also seen the Grace Hopper conference feature more male keynote speakers, which is problematic when the goal of the conference is to allow women to connect to role models that look like them.

When I did a bit of research for this blog post, I looked at the board member list for Anita Borg Institute, who organizes the Grace Hopper Conference. I was unsurprised to see major corporate executives hold the majority of Anita Borg Institute board seats. However, I was curious why the board member page had no pictures on it. I used Google Image search in combination with the board member’s name and company to create this image:
anita-borg-board

My unease was recently echoed by Cate Huston, who also noticed the trend towards corporations trying to co-opt women’s only spaces to feed women into their toxic hiring pipeline. Last week, I also found this excellent article on white feminism, and how white women need to allow people of color to speak up about the problematic aspects of women-only spaces. There was also an interesting article last week about how “women’s only spaces” can be problematic for trans women to navigate if they don’t “pass” the white-centric standard of female beauty. The article also discusses that by promoting women-only spaces as “safe”, we are unintentially promoting the assumption that women can’t be predators, unconsciously sending the message to victims of violent or abusive women that they should remain silent about their abuse.

So how to do we expand women-only spaces to be more inclusive, and move beyond white corporate feminism? It starts with recognizing the problem often lies with the white women who start initiatives, and fail to bring in partners who are people of color. We also need to find ways to fund inclusive spaces and diversity efforts without big corporate backers.

We also need to take a critical look at how well-meaning diversity efforts often center around improving tech for white women. When you hear a white male say, “We need more women in X community,” take a moment to question them on why they’re so focused on women and not also bringing in more people of color, people who are differently abled, or LGBTQ people. We need to figure out how to expand the conversation beyond white women in tech, both in external conversations, and in our own projects.

One of the projects I volunteer for is Outreachy, a three-month paid internship program to increase diversity in open source. In 2011, the coordinators were told the language around encouraging “only women” to apply wasn’t trans-inclusive, so they changed the application requirements to clarify the program was open to both cis and trans women. In 2013, they clarified that Outreachy was also open to trans men and gender queer people. Last year, we wanted to open the program to men who were traditionally underrepresented in tech. After taking a long hard look at the statistics, we expanded the program to include all people in the U.S. who are Black/African American, Hispanic/Latin@, American Indian, Alaska Native, Native Hawaiian, or Pacific Islander. We want to expand the program to additional people who are underrepresented in tech in other countries, so please contact us if you have good sources of diversity data for your country.

But most importantly, white people need to learn to listen to people of color instead of being a “white savior”. We need to believe people of color’s lived experience, amplify their voices when people of color tell us they feel isolated in tech, and stop insisting “not all white women” when people of color critique a problematic aspect of the feminist community.

Trying to move into more intersectional feminism is one of my goals, which is why I’m really excited to speak at the Richard Tapia Celebration of Diversity in Computing. I hadn’t heard of it until about a year ago (probably because they have less corporate sponsorship and less marketing), but it’s been described to me as “Grace Hopper for people of color”. I’m excited to talk to people about open source and Outreachy, but most importantly, I want to go and listen to people who have lived experiences that are different from mine, so I can promote their voices.

If you can kick in a couple dollars a month to help me cover costs for the conference, please donate on my Patreon. I’ll be writing about the people I meet at Tapia on my blog, so look for a follow-up post in late September!

First timer’s guide to FOSS conferences

AdaCamp Model Release Agreers

I’ve been going to FOSS conferences since 2006. My first open source conference was FreedomHEC in Seattle, a little 30-person conference for Linux users to protest Microsoft’s WinHEC. My next open source conference was OSCON, which had over a thousand attendees. They were both very different conferences, and as a college student, I really didn’t know what to expect. Going to your first open source conference can be intimidating, so I’ve complied ten tips for people who are new to the conference circuit.

Rule #1: The Hallway Track Matters Most

Conference talks are great because they teach you new skills or give you ideas. However, what conference talks are really for is giving you additional topics of conversation to chat with your fellow conference goers with. Hanging out after a talk ends to chat with the speaker is a great way to connect with speakers or fellow attendees that are passionate about a particular subject.

Why do I downplay conference talks? Because it’s really the networking at a conference that matters. The “hallway track” is where you’ll talk to fellow open source contributors, find your next job, or get excited about a new project. If you were only here for the talks, you would save yourself the travel expenses and sit in bed watching the livestream of the talks. So your first focus should be on the people at the conference.

There’s a lot of ways to maximize the networking you do. One small trick is to make sure you have a pen and physical paper cards. Moo cards are great, because you can add pictures to the front of the card. Maybe you want to add a picture of your latest circuit layout. Or travel pictures, so you can have an additional conversation starter. However, I would avoid putting a picture of yourself in a business suit on the front. I’ve seen one card that used that trick, and while it was great for identifying the person I talked to, it felt really pretentious.

When you receive someone’s business card, write down the reason you need to get a hold of them on the card, and any identifying information you need to remember who they are. You’ll thank yourself two weeks later when you recover from con fatigue and find a stack of forgotten business cards. I’ve tried using digital barcodes on my phone to redirect people to a business card page on my blog, but it takes too long for people to fiddle with their phones and it feels really awkward. Paper cards are old school, but they work best. I’ve also seen some hardware hackers print circuit board business cards. Those are really neat and memorable, but cost more and don’t allow people to write on them.

To optimize your networking, follow the one to many rule. If you mostly hang out with groups of people you know, you won’t end up making many new contacts. What you really want to find is a group of people that has one or two people you know, who can introduce you around. If you’re already in a group, face your body slightly outwards and a bit away from the person next to you, leaving the group open to new members.

If you don’t know anyone, you can stand around the coffee/snack/water table and drink/eat while you see if there are interesting conversations you can insert yourself into. Moving into a group of chairs and asking someone nearby if the chair is taken can be a good way to try and start a conversation about how the conference is going. If you find out someone is a speaker, you have an automatic topic of conversation later when you ask them how their talk went. Approaching a stranger with the question, “I love that sticker on your laptop, where did you get it?” is always a good way to get someone to talk about something they’re passionate about.

Rule #2: It’s OK to Skip a Session

Maybe you’re having an awesome conversation. Maybe you skipped lunch and you desperately need food. Or you’re speaking later in the day and you want to go over your slides one more time. It’s fine to skip a session. I promise you won’t miss out on much.

Rule #3: Triple Conflict Sessions Are Common

Conference schedules are a feast or famine affair. There’s usually at least two sessions where I’m not interested in any of the talks, and two more sessions where there are three talks I want to attend. Scheduling is the bane of conference organizer’s existence (it’s an NP complete problem!) so cut them some slack.

When I am triple booked, I’ll go to the talk on a subject I’m least familiar about. Conferences are a place to learn, and a place to network with experts, so it’s better to go to a talk on a new topic you’re interested in. The only exception I make with this rule is if the triple booked session is in the late afternoon. At that point, my brain is full of new knowledge, and it’s better if I watch the new topic talk as a video later.

Rule #4: Care and Feeding of Introverts

Most conferences start around 9am or 10am, include a break for lunch (either provided by the conference, or on your own at local restaurants), continue until 5pm or 6pm, and then have evening events from 7pm until around 10pm or 11pm. A lot of conference goers then go drink and chat until the wee hours of the morning.

I know a lot of geeks who don’t attend big parties at home, but get all their socialization in during the conference season. It can be exhausting as an introvert to attend a conference. It’s not just that there’s a whole lot of people, it’s that there’s a whole lot of strangers, and that can trigger social anxiety.

If you’re an introvert (like me), there’s a couple of different ways you can take care of yourself. Personally, my social batteries get pretty low around mid-afternoon. If I don’t recharge, I know I’ll be a untalkative zombie during the evening social events. Every day, when I’m planning out my conference schedule, I deliberately pick one uninteresting afternoon session to go back to my hotel room and take a nap. If my hotel room is too far, I carry a headband to cover my eyes and unabashedly take a nap on a conference bench. Those naps have seriously saved my life.

Evening events are always a game of chance for me. As an introvert, the best evening events for me are ones that center around an activity. Some of my favorite evening events at conferences have been bowling, wandering around a museum, playing arcade games and DDR, going for a boat ride, and playing board games. My least favorite conference evening events are the booze-filled bar crawls. When my anxiety at being crushed in a giant crowd of strangers is coupled with a throng of drunk dude-bros that are shouting at each other over the loud music, I last about an hour before I flee. It’s perfectly fine to retreat back to your hotel room. It’s also acceptable to find fun people to flee with, or even arrange to meet said fun people elsewhere and avoid the crowds.

Rule #5: Care and Feeding. Period.

When I first started going to conference, I wanted to see all the talks, talk to all the people, and do all the events. However, a lot of conferences don’t allow enough time at lunch to get food from restaurants. If you want to make the sessions before and after lunch, that typically means eating some crappy food at the convention center, and rushing off without talking to people. Rule #1 applies, so what you really want to do is be at peace with coming late into the afternoon talks, and wander off with some interesting people to chat over a leisurely, healthier lunch.

I’m not going to lie, a lot of people at conferences drink at evening events. Some of them don’t, and honestly, some of the most fun I’ve had has been hanging out and bonding with other non-drinkers. Many geeks don’t drink all that much at home, but tend to throw back A LOT of drinks when they’re paid for out of some start up or big company’s budget instead of their own pocket. If you don’t feel comfortable in that situation, make sure you come with a buddy or another non-drinker.

If you do drink, take care of yourself. Drink water, take some vitamins, and have a plan for getting back to your hotel in a foreign city. Due to past experience with the morning after impacts of heavy drinking, I always pack an extra pair of jeans. They add weight to your suitcase, but having an additional pair of pants is better than washing vomit out of your only pair of pants and then ironing them dry before you rush off to your flight. Did I mention I don’t drink much any more?

The morning after evening events are brutal. Not gonna lie, I’ve never made breakfast plans at a conference. I do often find people to eat with in the hotel breakfast buffet, but I never make firm plans because I don’t know how long I need to sleep. Sometimes I skip the hotel breakfast and just head straight to the conference. On those days, I’m really glad I packed myself breakfast bars.

Basically, you need to be a soccer mom to yourself. Pack yourself a water bottle and food, and maybe some fruit snacks and animal crackers. 😉

Rule #6: Trust the Whisper Network

Conferences are full of people who are interesting, and occasionally house people who are known harassers. As a newcomer to a conference, you’re likely to be targeted because you don’t know who those harassers are. Trust your gut if something feels off. Recognize when someone is trying to isolate you, test your boundaries, or ply you with alcohol. Know whether your conference has a Code of Conduct, and program in the email or phone number of the CoC contact into your phone, along with the local police non-emergency numbers. Figure out how to identify conference organizers, and make friends with minorities and women who have been to the conference in the past. It’s likely they will know who you should avoid, even if they can’t publicly talk about it because they might get sued for libel.

Rule #7: Know Your Conference Perks

Did you know that most conferences have a travel budget? That’s right, if you’re a speaker or a student or a member of an underrepresented group in open source, you may be able to get your conference travel paid for. Organizers will usually pay for conference fees, sometimes pay for airfare (it may be harder to get them to pay for international travel), and may pay for hotel and hood. If you’re a well-recognized speaker who has been invited to give a talk or keynote, organizers may also pay you a stipend (the amount of which is usually negotiable).

Some (but not many) conferences have consuites where any attendee can get free food. Others provide lunch. Some conferences have evening events with free food. If you’re a speaker, find the speaker’s lounge, which usually has coffee and cookies and other interesting speakers to talk to.

Rule #8: Swag is a Double-edged Sword

Do you really need another beer bottle opener? Are you willing to lug home a dozen t-shirts that don’t fit? When someone asks what that sticker is on your laptop, will you lose geek cred because you slapped some company’s brand on your property without actually using their product?

The downside of all that free stuff is that it’s actually a ploy to provide companies with your contact information. A new trick for conferences is to include a bar code on the badge with your personal information on it. Booth wranglers will scan your badge, and do whatever they want with that information. Give it to recruiters, sign you up for their email lists, or even give that information to third parties. Do you know what’s on your badge bar code? Could a stalker glean information like your phone number and zip code from it? You can always choose to take a pen to the bar code, and booth people should still give you all the freebies and manually sign you up for entry into any giveaways.

Rule #9: Follow up

You’ve met a bunch of awesome people, gathered a stack of business cards, and gotten in contact with recruiters at a fascinating new company. After the conference is over, it’s time to follow up. It’s best to wait at least a couple days, so that people traveling can get back home and start shoveling through their overflowing inboxes. Waiting more than two weeks is probably not the best practice for most conference contacts, but that may be an appropriate amount of time if you’re following up with someone you have given your resume to.

Rule #10: Project Passion Explosion

Conferences are a slew of new ideas. You’re likely to be introduced to a whole bunch of new projects and interesting people. The best conferences are the ones where you feel like your brain is on fire with all the new projects you want to do. However, be reasonable about your time commitments. Don’t try to start more than one project after a convention. Wait a week or two and see if you’re still passionate about the project before you start on it. Unless it’s a time machine. Please start on that right away, because every conference goer needs one.

Graphics linkspam: A maze of twisty passages

One of the things I’ve been frustrated with lately is an overview document of Intel graphics architecture. There’s the massive Intel graphics Programmer Reference Manuals (which are so big they need to be separate PDFs for each chapter), but if you’re skimming each chapter intro for an overview, you’re likely to get lost. Fortunately, I stumbled across a 22-page Intel graphics architecture overview document that looks much more promising.

Another cool find this week was two sites that document which OpenGL extensions hardware manufacturers implement on Linux. If you’re a game designer looking to improve performance by using the latest wiz-bang extension, it’s important to know if that extension is actually implemented in all the hardware you care about. There’s the Mesa Matrix site that says whether a particular hardware vendor implements an extension, and another site that further breaks it down per-platform.

Code of Conduct Warning Signs

I’ve got something on my chest that needs to be expressed. It’s likely to be a bit ranty, because I’ve got some scars around dealing with this issue. I want to talk about Codes of Conduct (CoCs).

No Trespassing!

Over the last five years, I’ve watched the uptick in adoption of CoCs in open source conferences. I’ve watched conferences try to adopt a CoC and fall completely flat on their face because they completely misunderstood the needs of minorities at their conferences. In recent years, I’ve watched open source communities start to adopt CoCs. For some communities, a CoC is an after thought, a by-product of community leadership stepping up in many different ways to increase diversity in open source.

However, a worrysome trend is happening: I see communities starting to adopt Codes of Conduct without thinking through the implications of them. A CoC has become a diversity checkmark.

Why is this? Perhaps it’s because stories of harassment has become wide spread. People look at the abuse that G4mer Goobers have thrown at women developers, especially women of color and trans women, and they say, “I don’t want those types of people in my community.” For them, a Code of Conduct has become a “No Trespassing” sign for external harassers.

In general, that’s fine. It’s good to stand up to harassers and say, “That’s not acceptable.” People hope that adding a Code of Conduct is like showing garlic to a vampire: they’ll hiss and run off into the darkness.

Pot, meet Kettle

However, a lot of people who are gung-ho about banning anonymous online harassers are often reluctant to clean their own house. They make excuses for the long-standing harassers in their community, and they have no idea how they would even enforce a CoC against someone who is an entrenched member of the community. Someone who organizes conferences. Someone who is a prolific reviewer. Someone who is your friend, your colleague, your drinking buddy.

You see, no one wants to admit that they are “that person”. It’s hard to accept that everyone, including your friends, are unconsciously biased. It’s even harder to admit that your friends are slightly racist/homophobic/transphobic/etc. No one wants to recognize the ablist language they use in their every day life, like “lame”, “dumb”, or “retarded”. It’s tough to admit that your conference speakers are mostly cis white men because you have failed to network with minorities. It’s difficult to come to grips with the fact that your leadership is toxic. It’s embarrassing to admit that you may be too privileged and so lacking in understanding of minorities’ lived experiences that you may need to reach outside your network to find people to help you deal with Code of Conduct incidents.

Code of Conduct Enforcement

And you will have incidents. People will report Code of Conduct violations. The important question is, how will you handle those incidents and enforce your CoC? You’ve put a “No Trespassing” sign up, but are you willing to escort people out of your community? Take their commit access away? Ask them to take a break from the mailing list? If you don’t decide up front how you’re going to enforce your Code of Conduct, you’re going to apply it unfairly. You’ll give your buddy a break, make excuses like, “But I know they’ve been working on that,” or, “Oh, yeah, that’s just so-and-so, they don’t mean that!”

You need to decide how you’ll enforce a Code of Conduct, and find diverse leadership to help you evaluate CoC violations. And for the love of $deity, if the minorities and louder allies on your enforcement committee say something is a problem, believe them!

Let’s fork it!

Another worrisome trend I see is that the people working on creating Codes of Conduct are not talking to each other. There is so much experience in the open source community leadership in enforcing Codes of Conduct, but it’s become a bike shed issue. Communities without experience in CoC enforcement are saying, “I’ll cherry-pick this clause from this CoC, and we’ll drop that clause because it doesn’t make sense for our community.”

We don’t write legal agreements without expert help. We don’t write our own open source licenses. We don’t roll our own cryptography without expert advice. We shouldn’t roll our own Code of Conduct.

Why? Because if we roll our own Code of Conduct without expert help, it creates a false sense of security. Minorities who rely on a Code of Conduct to grant them safety in an open source community will get hurt. If leadership is implementing a Code of Conduct as a diversity check mark, it papers over the real problem of a community that is unwilling to put energy into being inclusive.

Diversity Check Mark Complete!

I also see smaller communities scrambling to get something, anything, in place to express that they’re a safe community. So they take a standard Code of Conduct and slap it into place, without modifying it to express their communities’ needs. They don’t think about what behaviors they want to encourage in order to make their community a safe place to learn, create, and grow. They don’t think about how they could attract and retain diverse contributors (hint, I recently talked about some ideas on that front). They don’t think about the steps that they as leaders need to take in order to expand their understanding of minorities’ lived experiences, so that they can create a more inclusive community. They don’t think about the positive behaviors they want to see in their community members.

When I see an unmodified version of a Code of Conduct template in a community, I know the leadership has put up the “No Trespassing” sign to stop external harassers from coming in. But that doesn’t mean the community is inclusive or diverse. It could be a walled garden, with barriers to entry so high that only white men with unlimited amounts of spare time and a network of resources to help them can get inside. It could be a barb-wire fence community with known harassers lurking inside. Or it could be a community that simply found another CoC was good enough for them. I can’t know the difference.

Ask for Expert Advice

My take away here is that implementing a Code of Conduct is a hard, long, process of cultural change that requires buy-in from the leadership in your community. Instead of having an all-out bike-shed thread on implementing a CoC, where people cherry-pick legal language without understanding the implementation details of removing that language, go talk with an expert. Safety First PDX, Ashe Dryden, and Frame Shift Consulting are happy to provide consulting, for a fee. If you don’t have money to pay them (and you should pay women for the emotional labor they do to create welcoming communities!), then you’ll need to spend a bunch of time educating yourself.

Read *everything* that Safety First PDX has to say about Code of Conduct design and enforcement. Read the HOW-TO design a Code of Conduct post on the Ada Initiative website. Watch Audrey Eschright talk about Code of Conduct enforcement. Look at the community code of conduct list on the Geek Feminism wiki. These are all a long reads, but these are known experts in the field who are offering their expertise to keep our open source communities safe.

In Conclusion

Don’t roll your own Code of Conduct without expert advice. You wouldn’t roll your own cryptography. At the same time, don’t make a Code of Conduct into a check mark.

SCALE: Improving Diversity with Maslow’s hierarchy

If you’re reading this, I’ve just given my keynote at the Southern CAlifornia Linux Expo (SCALE). This post serves to link to all the different studies and resources I talked about, as well as my slides with notes and photo attributions can be found here.

Here’s the video of the keynote:

Introduction

In open source, we talk about meritocracy a lot. However, emphasizing meritocracy can actually lead to more bias against women:

It’s much better to actually acknowledge our biases and privilege. If you haven’t read it yet, Scalzi’s essay “White Straight Male is the Lowest Difficulty Setting” is an excellent starting point for exploring the impact of privilege.

There are lots of different ways individuals can contribute to improving diversity. However, this talk focuses on the macro view of how we as open source community individuals can increase diversity.

The inspiration for the structure of this talk came from Abraham Maslow’s paper “A theory of human motivation.”

Level 1: Geek Homeostatis, or Deep Hack Mode

Maslow talked about the needs that are required to keep the body’s chemical systems functioning (homeostatis). Those needs include warmth, food, water, and excretion.

Trans people often face harassment that keeps them from being able to use restrooms without fear. In response, an open source project called Refuge Restroom was created to map safe and gender-neutral restrooms. The project has also been featured in hack sessions by TransH4CK, an organization dedicated to developing open source projects that help trans*, gender non conforming, agender and non binary people.

What does homeostatis mean for a geek? I posit that it means deep hack mode, that magical time when you can pour everything into your project. But what is required in order to get into deep hack mode?

The first requirement is uninterrupted time. In open source, we expect people to volunteer their unpaid time to further the goals of our software. As Ashe Dryden explains, not everyone has the privilege to dedicate a lot of time to open source. Carers of young children and elderly relatives have very little time to spare. Women and especially women of color are more likely to be carers. Next time a candidate doesn’t have many github contributions, consider whether they’re a minority who may be a carer with little time, instead of someone who simply doesn’t want to contribute to open source.

There are some companies that are trying to help techie moms. The Paypal Recharge program provides job opportunities for women who have taken a break from their career (perhaps to raise a child) and want to switch into a technology career. Etsy is piloting providing executive coaching to new mothers. There’s also the non-profit Mother Coders, which helps women with children get into technology careers.

The next thing you need for deep hack mode is some sort of computer. Even with computer ownership increasing dramatically in the United States, the census data shows a deep disparity in who gets to own computers. African American and Hispanics, along with people in low and middle income brackets are far less likely to own a computer.

Women are also less likely to own a computer until all the men in the household have a computer. The book Women Don’t Ask talks about a series of interviews with computer science students at Carnegie Mellon University. They found that many female students didn’t start programming until college because their male siblings were granted more time on the computer or kept the computer in their room.

Even then, we have to consider what kind of computer minorities and lower class people are likely to buy, if they don’t buy a used computer. There are a whole bunch of different non-profits that are working to improve access to computers in low income communities around the globe. Free Geek operates in the United States (Portland and Chicago). Kids on Computers and Computer Reach gives computers internationally. The Raspberry Pi organization is also working to provide cheap computers globally. Why not donate your old computer, or donate money to refurbish old systems or make new cheap ones?

Of course, you need electricity to power your computer, and you need internet access to participate in open source communities. According to the 2011 Indian Census, only 67% of households had electricity. This Chrome Dev Summit talk on India’s internet growth was very enlightening. Most of India (19% or 248.53 million people) are on mobile networks, usually on a 2G network with an average speed of 60 KB/second. It costs them around 17 hours of minimum wage pay to have a 500 MB data plan. This means Indian internet users are very bandwidth conscientious. As community members of open source web tools and  services, we have an obligation to help lower our bandwidth footprint.

Level 2: Security

I talked at length about the Petrie multiplier, which proves that an attack on sexism is not an attack on men. I also talked about the requirements for an effective Code of Conduct, and how rolling your own code of conduct without listening to inputs is like rolling your own cryptography. Instead of making shit up and having it blow up in your face, you could pay women in tech who already have experience in improving diversity. I recommend contacting Ashe Dryden, Frame Shift Consulting, or Safety First PDX. *Edit: As of January 2017, I also offer diversity consulting through Otter Tech*

Level 5: Self Actualization

I mentioned a whole bunch of cool people in my final slides, and the diversity programs they work on:

There’s a lot of awesome projects that you can contribute to. If you don’t have time, I recommend donating to the Software Freedom Conservancy and Outreachy.

Metrics of Haters

When I posted my Closing a Door post, I mentioned that a team of moderators would be filtering comments for me. Comments that did not meet my comment policy would not be approved. Moderators also found that some comments simply did not further the conversation, were unclear and confusing due to translation issues, or were just contentless spews of hatred.

The comments on that post are now closed. The moderators approved a total of 254 comments, with 213 comments on my “Closing a Door” post, and 39 comments on my follow-up post “What Makes A Good Community?” The moderators also filtered out 186 comments total on those two posts. Now that the internet shit storm is over, I thought it would be interesting to take a peek into the acid-filled well in order to pull out some metrics.

Of course, I didn’t want to actually read the comments. That be silly! It would completely defeat the purpose of having comment moderators and let the trolls win. So, instead I used the power of open source to generate the metrics.  I used the WordPress Exporter plugin to export all the comments on the two posts in XML. Then I used the python wpparser library to parse the XML into something sensible. From there, the program wrote the commenters’ names, email addresses, and IP addresses [1] into a CSV. I did some manual categorization of that information in Google docs.

Repeat Offenders or Drive-by Haters?

70% of the 186 filtered comments were from unique IP addresses. The remaining 30% of comments were generated by 19 different people, who left an average of three comments each. The most persistant troll commented 10 times.

Anonymous Cowards or Brave Truth Tellers?

72% of the 186 comments did not include a full name. Of the commenters that did not include a full name:

  • 39 people used just a first name, making up 24% of the comments.
  • 25 people used what looks like internet nicks, accounting for 16% of the comments.
  • 17 people used various forms of the word “anonymous” in the name field, making up 9% of the comments.
  • 12 people used an English word instead of a name, accounting for 8% of the comments.
  • 4 people used obviously fake names, accounting for 7% of the comments.
  • 8 people used their initials or one letter, accounting for 5% of the comments.
  • 5 people used a slur in their name, accounting for 3% of the comments.
  • 2 people used a threat in their name, accounting for 1% of the comments. [Edit: make that 3, or 2%]

Community Members or Internet Trolls?

38 people used a full name, accounting for 28% of the comments. That means approximately 1/3 were brave enough to put their real name behind their comments. (Or a full fake name.) The question becomes, are these people actually a part of the open source community? Are they people who have actually interacted on an open source mailing list before? To answer these questions, I choose to search the author name in the Mailing List Archives (MARC) where a variety of open source mailing lists are archived, including the Linux kernel subsystem mailing lists, BSD, database lists, etc.

Of the 38 people who used their real name, 14 people had interacted on an open source mailing list archived by MARC. They made up 8% of the filtered comments. Ten of those people had more than 10 mails to the lists.

[Edit] Of the 25 people that used what looked like internet nicks, 11 of them may be open source users (see analysis below in the comments). That accounted for 8% of the filtered comments.

The important take away here is that only 16% of the filtered comments were made by open source users and developers. This is an important finding, since the article itself was about open source community dynamics.

[1] Before you scream about privacy, note that my comment policy allows me to collect and potentially publish this information.

Building a custom Intel graphics stack

When I worked as a Linux kernel developer, I often ran across people who were very concerned about compiling and installing a custom kernel. They really didn’t like running the bleeding edge kernel in order to check that a specific bug still existed. Who can blame them? A rogue kernel can corrupt your file system and you could lose data.

Fortunately, there is a safer way to build the latest version of drm and Mesa in order to check if an Intel bug still exists in the master branch. Since Mesa is just a userspace program, it is possible to install it to a custom directory, set a lot of environment variables right, and then your programs will dynamically link against that custom drm and Mesa binaries. Your desktop programs will still run under your distro’s system-installed mesa version, but you can run other programs linked against your custom mesa.

Unfortunately, mesa has some dependencies, and the instructions for how to build into a custom directory are kind of scattered all over the mesa homepage, the DRI wiki, the Xserver wiki, github, and mailing list posts. I’m going to attempt to condense these instructions into one single place, and then clean up those pages to be consistent later.

Debug build or better performance?

In this tutorial, I’ll assume that you want to build a version of drm and mesa with debugging enabled. This *will* slow down performance, but it will enable you to get backtraces, run gdb, and gather more debugging information than you normally would. If you don’t want a debug build, remove the parts of the commands that add the “debug” flag to the USE environment variable or config.mk files.

The point of this tutorial is to be able to install drm and mesa in a directory, so that you don’t have to install them over your distro’s binaries. This means you’ll be able to run the specific test you need, without running into other bugs by running your full desktop environment on the bleeding edge graphics stack. In this tutorial, I will assume you want to put your graphics installation in $HOME/graphics-install. Change that to whatever your heart’s desire is.

If you are working behind a proxy, you’ll need to have a .gitconfig file in your homedir that tells git how to clone through the proxy.

I also assume you’re running a debian-based system, specifically Ubuntu 14.04 in my case. If you’re on an RPM-based distro, change the package install commands accordingly.

Get mesa dependencies

sudo apt-get build-dep libdrm mesa mesa-utils
sudo apt-get install linux-headers-`uname -r` \
    libxi-dev libxmu-dev x11proto-xf86vidmode-dev \
    xutils-dev mesa-utils llvm git autoconf automake \
    libtool ninja-build libgbm-dev

Clone the repositories

mkdir git; cd git
git clone git://anongit.freedesktop.org/git/mesa/mesa
git clone git://anongit.freedesktop.org/mesa/drm
git clone git://github.com/chadversary/dev-tools.git
git clone git://anongit.freedesktop.org/piglit
git clone git://github.com/waffle-gl/waffle.git
git clone git://anongit.freedesktop.org/mesa/demos.git

Set up Chad’s development tools

Chad Versace has been working on a set of scripts that will set up all the right environment variables to run programs that will use custom-installed mesa and drm binaries. Let’s get those configured properly.

Edit your .bashrc to include the following lines:

export GOPATH=$HOME/go
export PATH=/usr/lib/ccache:$HOME/bin:$PATH:$GOPATH/bin
export PYTHONPATH=~/bin/mesa-dev-tools/bin/:$PYTHONPATH

Now it’s time to set up Chad’s tools.

cd dev-tools/

We’ll be installing everything in ~/graphics-install, so we need to create a config.mk file with this contents:

prefix := $(HOME)/graphics-install
USE := "debug"

This will add the debug flag to all builds, which will add symbols so you can use gdb (as well as add some additional code that could impact performance, so don’t add the flag if you’re doing performance testing!).

Build and install the development scripts:

make && make install

Exit your current shell, and start a new shell, so that the changes to the .bashrc and the installation of Chad’s scripts take effect.

Next, we need to get all the paths set properly to use Chad’s scripts to build mesa and libdrm into ~/graphics-install. We invoke the prefix-env script, and tell it to exec the command to start a new shell:

cd git/dev-tools
PREFIX="$HOME/graphics-install" USE="debug" \
    bin/prefix-env exec --prefix=$HOME/graphics-install bash

Double check that worked by seeing whether we have the right mesa-configure script on our path:

sarah@dingo:~/git/dev-tools$ which mesa-configure
/home/sarah/graphics-install/bin/mesa-configure

Check which Mesa version you’re running. Later, after installing a custom Mesa, we’ll verify the installation by confirming that the active Mesa version has changed.

sudo glxinfo > /tmp/glxinfo-old.txt

Note that glxinfo calls through the Xserver to get the information for what Mesa we’re using. Note that if your system xorg installation is too old, the Xserver won’t be able to find an API-compatible version of Mesa, and you’ll see errors like:

Error: "couldn't find RGB GLX visual or fbconfig"

Fortunately, we can run many Mesa programs without involving the Xserver. Another way to find what version of mesa you’re running without going through the Xserver is to use wflinfo command:

sudo wflinfo --platform gbm --api gl > /tmp/wflinfo-old.txt

We can see which version of mesa (11.0.2) is installed by default on Ubuntu 14.04:

sarah@dingo:~/git/dev-tools$ grep Mesa /tmp/*info-old.txt
client glx vendor string: Mesa Project and SGI
OpenGL core profile version string: 3.3 (Core Profile) Mesa 11.0.2
OpenGL version string: 3.0 Mesa 11.0.2
OpenGL ES profile version string: OpenGL ES 3.0 Mesa 11.0.2

Now, that we have Chad’s tools set up, and we’ve set up the environment variables, it’s important to build everything from the shell where you ran the prefix-env command. If you need to open up additional virtual consoles, make sure to change into ~/git/dev-tools/ and re-run the prefix-env command.

Building libdrm

The direct rendering manager library, libdrm, is a prerequistite for mesa. The two projects are pretty intertwined, so you need to have updated installations of both.

Change directories into your libdrm repository, and configure libdrm with the libdrm-configure script (note that PREFIX was already set when we exec’ed with the prefix-env script):

USE="debug" libdrm-configure
make && make install

Building Mesa

Change directories into your mesa repository, and configure mesa with the mesa-configure script:

cd ../mesa
USE="debug" mesa-configure
make && make install

Building Waffle

Waffle is a library for selecting an OpenGL API and window system at runtime.

cd ../waffle
USE=debug waffle-configure
ninja && ninja install

For some reason, waffle is different from all the other projects, and likes to install libraries into $PREFIX/lib/x86_64-linux-gnu/ If you’re on a debian-based system, you may have to change the configuration files, or simply move the libraries one directory down.

Building glxinfo

Confusingly, a useful debugging tool like glxinfo is found in a mesa repository named demos. Change into that directory:

cd ../demos

Since Chad’s tools don’t cover installation of the demo tools, we’ll have to configure them by hand:

autoreconf --verbose --install -s
./configure --prefix="$HOME/graphics-install"
make -j8 && make install

Confirm installation

Confirm that the environment’s Mesa version matches the version you installed. It should differ from the Mesa version we checked earlier.

sudo glxinfo > /tmp/glxinfo-new.txt

Or run wflinfo instead:

sudo wflinfo --platform gbm --api gl > /tmp/wflinfo-new.txt
grep Mesa /tmp/*info-new.txt

You should see something about a development version of mesa in the output.

Building Piglit

Piglit is the test infrastructure and tests for libdrm and mesa.  Let’s build it:

cd ../piglit
USE=debug piglit-configure

Piglit has a slightly different build system than drm, mesa, and waffle. After the first build, the dependencies for piglit tests means it takes a very long time to recompile the tests after a small change is made. This is due to the simplicity of cmake. Instead, it’s recommended to use the ninja build system with piglit.

Make piglit:

ninja

Install piglit:

ninja install

Run your tests

Anytime we want to use the newly installed mesa and drm, we need to rerun the prefix-env script to set up all the graphics environment variables to point to those binaries:

PREFIX="$HOME/graphics-install" USE="debug" \
    bin/prefix-env exec --prefix=$HOME/graphics-install bash

Since we haven’t compiled the full Xserver stack, we have to run piglit with a different platform than X11. If you run `piglit run –help`, you’ll see that a platform could be x11_egl, glx (which is actually calling the GLX api through the Xserver), mixed_glx_egl (which also implies going through the Xserver), wayland, and gbm. The most simple platform is gbm.

Here’s how you run your very first sanity test with gbm:

PIGLIT_PLATFORM=gbm ./piglit run \
    tests/sanity.tests results/sanity.results

If the output says you passed, give yourself a pat on the back! If not, you probably don’t have something installed correctly. You may want to exit all shells, run `git clean -dfx` (to clean out all tracked files) in all the repos, and try again.

To get a more detailed test report, you can run the `piglit summary` command with either console (for text output, good if you don’t have X running), or with html to generate pretty webpages for you to look at on another machine. Piglit will also output test results in a form Jenkins can use.

./piglit summary console results/sanity.results
./piglit summary html --overwrite summary/sanity results/sanity.results

You’ll need the overwrite or append flag if you’re writing results to the same directory.

There’s even more explanations of what you can do with piglit on these two blog posts.

Running games or benchmarks

Once you’ve run the prefix-env script, you should be able launch benchmarks or other tests. Running games or steam with a custom mesa installation is harder. Since most games are going to use the Xserver platform to call into Mesa’s GL or EGL API, you may need to compile a new Xserver as well.

Optional kernel installation

Sometimes you may want to run the bleeding-edge Intel graphics kernel. Confusingly, the kernel isn’t hosted on git.kernel.org! Use the drm-intel-nightly branch from the drm-intel repo on freedesktop.org:

git clone git://anongit.freedesktop.org/drm-intel

Instructions on compiling a custom kernel can be found here:

http://kernelnewbies.org/FirstKernelPatch

Additionally, you may need to set the i915 kernel module parameter to enable new hardware support.  You can do this by changing the line your grub configuration defaults in /etc/default/grub to this:

GRUB_CMDLINE_LINUX_DEFAULT="i915.preliminary_hw_support=1"

And then you’ll need to update your grub configuration files in /boot by running:

sudo update-grub

Graphics linkspam: Bugs, bugs, I’m covered in bugs!

Reporting bugs to Intel graphics developers (or any open source project) can be intimidating. You want the right developers to pay attention to your bug, so you need to provide enough information to help them classify the bug. Ian Romanick describes what makes a good Mesa bug report.

One of the things Ian talks about is tagging your bug report with the right Intel graphics code name, and providing PCI ID information for the graphics hardware. Chad Versace provides a tool to find out which Intel graphics you have on your system. That tool is also useful for translating the marketing names to code names and hardware details (like whether your system is a GT2 or GT3).

In the “omg, that’s epic” category, Adrian  analyzes the graphics techniques used in Grand Theft Auto V on PS3. It’s a great post with a lot of visuals. I love the discussion of deleting every-other-pixel to improve performance in one graphics stage, and then extrapolating them back later. It’s an example of something that’s probably really hardware specific, since Kristen Hogsberg mentioned he doesn’t think it will be much help on Intel graphics hardware. When game designers know they’re only selling into one platform, they can use hardware-specific techniques to improve graphics performance. However, it will bite them later if they try to port their game to other platforms.

How to approach a new system: Linux graphics and Mesa

By now, most of the Linux world knows I’ve stopped working on the Linux kernel and I’m starting work on the Linux graphics stack, specifically on the userspace graphics project called Mesa.

What is Mesa?

Like the Linux kernel, Mesa falls into the “operating system plumbing” category. You don’t notice it until it breaks. Or clogs, slowing down the rest of the system. Or you need plumbing for your shiny new hot tub, only you’re missing some new essential feature, like hot water or tessellation shaders.

The most challenging and exciting part of working on systems “plumbing” projects is optimization, which requires a deep understanding of both the hardware limitations below you in the stack, and the needs of the software layers above you. So where does Mesa fit into the Linux graphics stack?

Mesa is the most basic part of the userspace graphics stack, sitting above the Linux kernel graphics driver that handles command submission to the hardware, and below 3D libraries like qt, kde, or game engines like Unity or Unreal. A game engine creates a specialized 3D program, called a shader, that conforms to a particular rev of a graphics spec (such as EGL 3.0 or GLES 3.1). Mesa takes that shader program and compiles it into graphics hardware commands specific to the system Mesa is running on.

What’s cool about Mesa?

The exciting thing for me is the potential for optimizing graphics around system hardware limitations. For example, you can optimize the compiler to generate less graphics commands. In theory, less commands means less for the hardware to execute, and thus better graphics performance. However, if each of those commands are really expensive to run on your graphics hardware, that optimization can actually end up hurting your performance.

This kind of work is fun for me, because I get to touch and learn about hardware, without actually writing Verilog or VHDL. I get to make hardware do interesting things, like add a new features that makes the latest Steam game actually render or add an optimization that improves the performance of a new open source Indie game.

Understand how much you don’t know

Without knowledge of both the hardware and software layers above, it’s impossible to evaluate what optimizations are useful to pursue. For example, the first time I heard modern Intel GPUs have a completely separate cache from the CPU, I asked two questions: “What uses that cache?” and “What happens when the cache is full?” The hardware Execution Units (EUs) that execute graphics commands use the cache to store metadata called URBs. My next question, on discovering that URB size could vary, and that newer hardware had an ever-increasing maximum URB size, was to ask, “How does the graphics stack pick which URB size to use?” Obviously, picking a smaller URB size means that more URBs can fit in the cache, which means it’s less likely that an EU will have to stall until there’s room in the cache for the URB it needs for the set of graphics commands its executing. Picking an URB size that was too small would mean programs that use a lot of metadata wouldn’t be able to run.

Mesa is used by a lot of different programs with different needs, and each may require different URB sizes. The hardware has to be programmed with the maximum URB size, and changing that requires stopping any executing commands in the graphics pipeline. So you can’t go changing the URB size on the fly every time a new 3D graphics program starts running, or you would risk stalling the whole graphics stack. Imagine your entire desktop system freezing every time you launched a new Steam game.

Asking questions helps you ramp faster

It’s my experience with other operating system components that lead me to ask questions of my more experienced graphics developer co-workers. I really appreciate working in a community where I know I can ask these kinds of basic questions without fear of backlash at a newcomer. I love working on a team that chats about tech (and non tech!) over lunch. I love the easy access of the Intel graphics and DRI irc channels, where people are willing to answer simple questions like “How do I build the documentation?” (Which turns out to be complex when you run into a bug in doxygen that causes an infinite loop and increasing memory consumption until a less-than-capable build box starts to freeze under memory pressure. Imagine what would have happened if the experienced devs assumed I was a n00b and ignored me?)

My experience with other complex systems makes me understand that the deep, interesting problems can’t be approached without a long ramp up period and lots of smaller patches to gain understanding of the system. I do chafe a bit as I write those patches, knowing the interesting problems are out there, but I know I have to walk before I start running. In the meantime, I find joy in making blog posts about what I’m learning about the graphics pipeline, and I hope we can walk together.

What makes a good community?

*Pokes head in, sees comments are generally positive*

There’s been a lot of discussion in my comment sections (and on LWN) about what makes a good community, along with suggestions of welcoming open source communities to check out. Your hearts are in the right place, but I’ve never found an open source community that doesn’t need improvement. I’m quite happy to give the Xorg community a chance, mostly because I believe they’re starting from the right place for cultural change.

The thing is, reaching the goal of a diverse community is a step-by-step process. There are no shortcuts. Each step has to be complete before the next level of cultural change is effective. It’s also worth noting that each step along the way benefits all community members, not just diverse contributors.

Level 0: basic human decency

In order to attract diverse candidates, you need to be known as a welcoming community, with a clear set of agreed-upon social norms. It’s not good enough to have a code of conduct. Your leaders need to be actively behind it, and it needs to be enforced.

A level 0 welcoming community exhibits the following characteristics:

Level 1: on-boarding

The next phase in improving diversity is figuring out how to on-board newcomers. If diverse candidates are only 1-10% of newcomers, but you have a 90% fail rate for people who try to make their first contribution, well, you can’t expect many diverse newcomers to stick around, can you? It’s also essential to explain your unwritten tribal knowledge, so that diverse candidates (who are more likely to be afraid of upsetting the status quo) know what they’re getting into.

Signs of a level 1 welcoming community:

  • Documentation on where to interact with the community (irc, mailing list, bug tracker, etc)
  • In-person conferences to encourage networking with new members
  • Video or in-person chats to put a face to a name and encourage empathy and camaraderie
  • Documented first steps for compiling, running, testing, and polishing contributions
  • Easy, no-setup web harness for testing new contributions
  • Step-by-step tutorials, which are kept up-to-date
  • Coding style (what’s required and what’s optional, and who to listen to when developers disagree)
  • Release schedule and feature cut-off dates
  • How to give back non-code contributions (bug reports, docs, tutorials, testing, event planning, graphical design)

Level 2: meaningful contributions

The next step is figuring out what to do with these eager new diverse candidates. If they’ve made it this far through the gauntlet of toxic tech culture, they’re likely to be persistent, smart, and seeking a challenge. If you don’t have meaningful bigger projects for them to contribute to, they’ll move onto the next shiny thing.

Signs of a level 2 welcoming community:

  • Newbie todo lists
  • Larger, self-contained projects
  • Welcoming, available mentors
  • Programs to pay newbies (internships, summer of code, etc)
  • Contributors are thanked with heartfelt sincerity and an explicit acknowledgment of what was good and what could be improved
  • Community creates a casual feedback channel for generating ideas with newcomers (irc, mailing list, slack, whatever works)
  • Code of conduct encourages developers to assume good intent

Level 3: succession planning

The next step for a community is to figure out how to retain those diverse candidates. How do you promote these new, diverse voices in order to ensure they impact your community at a leadership level? If your leadership is stale, comprised of the same “usual faces”, people will leave when they start wanting to have more of a say in decisions. If your community sees bright diverse people quietly leave, you may need to focus on retention.

Signs of a level 3 welcoming community:

  • Reviewers are rewarded and questions from newcomers on unclear contributions are encouraged
  • Leaders and/or maintainers are rotated on a set time schedule
  • Vacations and leaves of absence are encouraged, so backup maintainers have a chance to learn new skills
  • Community members write tutorials on the art of patch review, release management, and the social side of software development
  • Mentorship for new presenters at conferences
  • Code of conduct encourages avoiding burnout, and encourages respect when people leave

Level 4: empathy and awareness

Once your focus on retention and avoiding developer burnout is in place, it’s time to tackle the task most geeks avoid: general social issues. Your leaders will have different opinions, as all healthy communities should! However, you need to take steps to ensure the loudest voice doesn’t always win by tiring people out, and that less prominent and minority voices are heard.

Signs of a level 4 welcoming community:

  • Equally values developers, bug reporters, and non-code contributors
  • Focuses on non-technical issues, including in-person discussions of cultural or political issues with a clear follow-up from leaders
  • Constantly improves documentation
  • Leadership shows the ability to recognize their mistakes and change when called out
  • Community manager actively enforces the code of conduct when appropriate
  • Code of conduct emphasizes listening to different perspectives

Level 5: diversity

Once you’ve finally got all that cultural change in place, you can work on actively seeking out more diverse voices and have a hope of retaining them.

Signs of a level 5 welcoming community:

  • Leadership gatherings include at least 30% new voices, and familiar voices are rotated in and out
  • People actively reach outside their network and the “usual faces” when searching for new leaders
  • Community participates in diversity programs
  • Diversity is not just a PR campaign – developers truly seek out different perspectives and try to understand their own privilege
  • Gender presentation is treated as a non-issue at conferences
  • Conferences include child care, clearly labeled veggie and non-veggie foods, and a clear event policy
  • Alcoholic drinks policy encourages participants to have fun, rather than get smashed
  • Code of conduct explicitly protects diverse developers, acknowledging the spectrum of privilege
  • Committee handling enforcement of the code of conduct includes diverse leaders from the community

The thing that frustrates me the most is when communities skip steps. “Hey, we have a code of conduct and child care, but known harassers are allowed at our conferences!” “We want to participate in a diversity program, but we don’t have any mentors and we have no idea what the contributor would work on long term!” So, get your basic cultural changes done first, please.

*pops back off the internet*

Edit: Please stop suggesting BSDs or Canonical/Ubuntu as “better” communities.