Outreachy Progress 2019-08

The majority of this month was spent herding coordinators to get communities signed up to participate in this Outreachy round. We now have 19 communities signed up to participate in the December 2019 to March 2020 round. Depending on community funding that’s still TBD, we could have somewhere between 31 and 44 interns this round. That’s on par with last round.

Organizer tasks this month:

  • Ping 25 communities who have participated in the past. 13 past communities have signed up this round.
  • Review 6 new communities who have never participated in Outreachy before
  • Meet with new potential sponsors
  • Provide on-boarding as time allows for new coordinators
  • Run two chats with Outreachy May 2019 interns
  • Review final feedback for Outreachy May 2019 interns
  • Review initial applications
  • Answer questions from applicants on Twitter and via email
  • Prep work for the Outreachy booth at the Tapia conference

Outreachy Progress 2019-07

The main focus of this month was gearing up for the application period for the December 2019 round. We’ve made some changes to the application process, which required both changes to the website and communication to stakeholders (applicants, coordinators, and mentors).

We aren’t forgetting the May 2019 interns though! Their internships are still active until August 20. We hosted two internship chats this month, and found a contractor to provide resume review for the interns.

Organizer work:

  • Coordinated with a new contractor to add Outreachy career advice services, including career chats and resume review
  • Reviewed mid-point feedback and facilitated conversations with mentors and interns
  • Coordinated with volunteers for the Outreachy Tapia booth about travel
  • Ordered Outreachy promotional materials for Tapia
  • Provided guidance to new communities thinking about participating in the December 2019 round

Documentation work:

  • Wrote a blog post explaining the process changes for applicants and mentors in the December 2019 application round
  • Updated Outreachy promotional materials with the new deadline changes
  • Wrote an Outreachy Applicant Guide
  • Started writing an Outreachy Internship Guide
  • Updated our mentor FAQ to mirror the process changes
  • Wrote an email to interns explaining how informational interviews work, and giving resume guidelines

Development work:

  • Deployed code to separate out the Outreachy initial application period and the contribution period
  • Deployed code to hide Outreachy internship project details until the contribution period opens
  • Deployed code to hide pending/approved status until the contribution period opens
  • Wrote 46 new tests for the deployed code, increasing the total number of tests to 77.

Outreachy progress 2019-06

Outreachy organizer admin tasks:

  • Run Outreachy biweekly internship chats
  • Read and follow up on initial feedback for Outreachy interns
  • Handle issues with intern time commitments
  • Remind Outreachy interns if they haven’t created a blog
  • Communicate with potential community and sponsors for the December 2019 round
  • Put out a call for Outreachy booth volunteers for the Tapia conference
  • Coordinate with potential contractors who can offer career advice and interviewing workshops to Outreachy interns.

Development tasks:

  • Work is progressing on the Outreachy website changes to separate out the initial application period and the contribution period.
  • Most of the template and view changes have been made.
  • Thorough tests need to be written next month to reveal any bugs.

Outreachy Progress 2019-05

Outreachy organizer admin tasks:

  • Announce the accepted Outreachy interns
  • Handle situations where interns were unable to accept the internship
  • Update the intern contract dates (we need to automate this in the website code!)
  • Send signed intern contracts to Conservancy
  • Send an invoice request to Conservancy for Outreachy sponsors
  • Follow all interns on Twitter, retweet their first blog posts if @outreachy is tagged
  • Run the first intern Q&A session

Development tasks:

  • Add website code to allow mentors to invite co-mentors and have co-mentors sign the mentorship agreement for any selected interns
  • Create page for organizers to see contact info for mentors who selected an intern (so we can easily subscribe them to the mentor mailing list)

Documentation

Outreachy at PyCon U.S. Sprints

We successfully tested the Outreachy website developer’s documentation at the PyCon U.S. sprints. This is the first step towards limiting the “bus factor” and ensuring that many people can work on the Outreachy website.

13 people participated in the sprints. Most were running Linux, but there was one Windows user who successfully followed our installation guide and successfully made their first contribution.

Half of the participants were unfamiliar with the Django web development framework that the Outreachy website uses. There were several people who had never made a contribution to free software before. We’re proud that they could make their first impact on the free software world with Outreachy!

Overall, 8 pull requests were merged, with 7 more pull requests waiting for review. The pull requests included improvements like clarifying our documentation, clarifying application questions, ensuring links on our opportunities page were valid, and improving the layout of our pages for past rounds.

Blog Post Prompts

During the Outreachy internship, interns are required to blog every two weeks. The Outreachy organizers found that interns often didn’t know what to blog about, so we started creating a series of blog post prompts. The prompts are highly relevant to the intern experience as the internship progresses.

Our first blog post prompts normalize the fact that all interns struggle during the first few weeks. The mid-point blog post prompt asks interns to reflect on their original project timeline, and how unexpected complexity means projects often have to be scaled back. We wanted to do a full series of blog post prompts last round, but we ran out of time before the next application period kicked off.

This round, we’re finishing out the last two blog post prompts for weeks 9 and 11 of the internship. They will focus on the next steps after the Outreachy internship, namely how interns can start a career in tech or free software.

The week 9 blog post prompt is for interns to write about what direction they would like to take their career. Some Outreachy interns are still in school, so we ask them to provide what time frame they want to take their next steps in.

The week 11 email will prompt the interns to work on their resumes, and then post them on their blog.

Career Development

Outreachy organizers are also in discussions with some contractors who may be able to provide some career advice to Outreachy interns. We’ve long wanted to provide more career services to interns, but haven’t been able to allocate organizer time to this. We’re still in negotiations, but we hope this round we can finally offer this.

Outreachy progress 2019-04

Outreachy organizer admin tasks:

  • Reviewed final feedback for all December 2018 Outreachy interns
  • Issued all December 2018 final stipend payment request emails
  • Reviewed all mentor intern selections for May 2019 to ensure only strong applicants were selected
  • Double checked some applicant time commitments via email
  • Resolved any intern selection conflicts between two Outreachy projects
  • Communicated with Google Summer of Code organization administrators when an applicant applied to both Outreachy and GSoC
  • Double checked that mentors who had selected multiple interns had a co-mentor to help them
  • Communicated with potential sponsors
  • Gathered billing information for all new sponsors
  • Prepped sponsor invoice drafts, thank you tweets, and ensured sponsor logo prominence
  • Discussed potential changes to the application period dates with Outreachy mentors

Development:

  • Triaged the Outreachy website GitHub issue tracker
  • Added a GitHub new project with newcomer-friendly issues in preparation for participating in the PyCon U.S. sprints
  • Added a new file with Python factories code to set up a local test database during specific periods in the internship round
  • Updated the Let’s Encrypt SSL certificate for the Outreachy Zulip Chat server
  • Fixed a number of bugs that mentors ran into during the intern selection process, or documented them for the PyCon U.S. sprints

Documentation:

  • Added 371 lines to the Outreachy website README.md and 5 new database graphics to help new developers working on the website
  • Started writing a manual documenting the steps I take as an Outreachy organizer each round.

Outreachy progress 2019-03

Summary of work:

  • Approved 1,013 initial applications total
  • Investigated an issue with KiwiIRC being banned on some IRC servers
  • Fixed some IRC link issues in the Outreachy website
  • Promoted new projects on Twitter and the Outreachy announce list
  • Sent semi-automated emails to remind applicants of the final application deadlines
  • Answered applicant’s questions, as time allowed
  • Communicated with potential Outreachy sponsors
  • Chased down some outstanding invoices from the December 2018 round
  • Communicated with December 2018 interns who had internship extensions

Outreachy progress: 2019-02

Summary of work this month:

  • Created final feedback form for interns and mentors
  • Contacted potential communities for the May to August 2019 round
  • Updated questions on the initial application form
  • Updated the website to the latest stable version of Django 1.11
  • Wrote a blog post announcing changes in eligibility criteria
  • Promotion on Twitter, emailing diversity in tech groups, job boards postings
  • Reviewed 874 initial application essays

The Outreachy internship program opened applications for the May to August round. Most of the time this month has been reviewing the 1,235 initial applications that have been submitted. ?

We’re definitely getting more applications this round. After the six week application period for the December to March round, we processed 1,817 initial applications. Less than two weeks into this round, we’ve had 1,235 initial applications submitted.

That sounds like a huge number, but that’s where the magic of Django comes in. Django allows us to collect time commitment information from all the applicants. We create a calendar of their time commitments and then see if they have 49 consecutive days free from full-time commitments from during the internship period.

So far, about 181 initial applications have been rejected because applicants had full-time commitments. (The number is usually higher in the December round because students in the northern hemisphere have a shorter break.)

We also check whether people are eligible to work in the countries they’re living in, whether people have participated in Outreachy or Google Summer of Code before, etc. There are 72 applications that were automatically denied because of those kinds of issues.

That leaves 982 applicants who were eligible for Outreachy so far. ? And we have to manually review every single applicant essay to see whether supporting this person would align with Outreachy’s program goal to support marginalized people in tech.

We ask specific essay questions to determine whether the applicant is underrepresented. We ask two more essay questions to determine whether they face discrimination or systemic bias in their learning environment or when looking for employment opportunities. Applicants have to demonstrate both characteristics. They have to be underrepresented *and* face discrimination.

It’s quite frankly difficult to spend 5-9 hours a day reading about the discrimination people face. We ask for personal stories, and people open up with some real horror stories. It’s probably re-traumatizing for them. It certainly impacts my mental health. Other people share less specific experiences with discrimination, which is also fine.

Sometimes reading essays introduces me to types of discrimination that are unfamiliar to me. For example, I’ve been reading more about the caste system in India and ethnic/tribal discrimination in Africa. Reading the essays can be a learning experience for me, and I’m glad we have multiple application reviewers from around the world.

One of the hardest things to do is to say no to an initial application.

Sometimes it’s clear from an essay that someone is from a group underrepresented in the technology industry of their country, but their learning environment is supportive and diverse, and they don’t think they’ll face discrimination in the workplace. Outreachy has to prioritize supporting marginalized people in tech, even if that means turning down underrepresented people who have the privilege to not face discrimination.

It’s also difficult because a lot of applicants who aren’t from groups underrepresented in tech equate hardship with discrimination. For example, a man being turned down for a job because they don’t have enough technical experience could be considered hardship. Interviewers assuming a woman doesn’t have technical experience because they’re a woman is discrimination. The end result is the same (you don’t get the job because the interviewer thinks you don’t have technical experience), but the cause (sexisim) is different.

Sometimes systemic issues are at play. For example, not having access to your college’s library because you have a mobility device and there’s no elevator is both discrimination and a systemic issue. Some communities face gender violence against women. The violence means parents don’t allow women to travel away to college, and some universities to restrict women to their dorms in the evenings. Imagine not being able to study after class, or not having internet in your dorms to do research. The reaction to these systemic issues incorrectly punish the people who are most likely to face harassment.

It’s frustrating to read about discrimination, but I hope that working with Outreachy mentors gives people an opportunity they wouldn’t otherwise have.

Outreachy Progress: 2019-01

Summary

  • Finished cleaning up the technical debt that kept us from having two Outreachy rounds active at once
  • Added code for gathering internship midpoint feedback
  • Migrated the travel stipend page off the old wiki for Outreachy to the Django website
  • Added a required field for mentors to provide the minimum computer system requirements to contribute to the project
  • Created intern blog post prompts for weeks 5 & 7
  • Followed up on all December 2018 sponsorship invoices

Minimum System Requirements

New for this Outreachy round is asking mentors to provide the minimum system requirements for their project. Many Outreachy applicants have second-hand, 10 year old systems. They may not have the memory to be able to run a virtualized development environment. In the past, we’ve had applicants who tried to follow installation instructions to complete their required contribution, only to have their systems hang.

By requiring mentors to provide minimum system requirements for their projects, we hope to help applicants who can’t afford a newer computer. We also hope that it will help communities think about how they can lower their technology barriers for applicants who face socioeconomic hardship

Simplifying Language

This month I migrated the travel stipend instructions page from our old wiki to the new travel page. During that migration, I noticed the language in the page was filled with complex vocabulary and longer sentences. That’s how I tend to write, but it’s harder for people who speak English as a second language to read.

I used the Hemmingway editor to cut down on complex sentences. I would recommend that people look at similar tools to simplify their language on their website

Debt, debt, and more technical debt

I had hoped that January would be spent contacting Outreachy communities to notify them of the round. Unfortunately, Outreachy website work took priority, as it wasn’t ready for us to accept community sign-ups.

Most of the work was done on cleaning up the technical debt I talked about in my last blog post. The website has to handle having two internship rounds active at once. For example, in January, mentors were submitting feedback for the December 2018 internships, while other mentors were submitting projects for the upcoming May 2019 internships.

A lot of the process was deciding how long to display information on the website. For example, when should mentors be able to choose an applicant as an intern for their project?

Mentors could find a potential candidate very early in the application period, so the very soonest they could choose an intern would be when the application period starts.

Most people might assume that interns can’t be selected after we announce the internships. However, in the past, interns have decided not to participate, so mentors have needed to select another applicant after the interns are announced. The very latest they could select an intern would be five weeks after the internships start, since we can’t extend an internship for more than five weeks.

It’s a complex process to decide these dates. It requires a lot of tribal knowledge of how the Outreachy internship processes work. I’m happy to finally document some of those assumptions into the Outreachy website code.

Outreachy Progress: 2018-12

One of my resolutions for 2019 is to be more transparent about the work I’ve been doing for Outreachy. Hopefully (fingers crossed) this means you’ll be seeing a blog post once a month.

I’ll also throw in a selfie per month. My face is changing since I’ve been on hormone replacement therapy (testosterone) for about 7 months now. I started to get some peach fuzz around month 5. It’s still patchy, but I’m growing it out anyway so I can see if I can get a beard!

New glasses too!

What is Outreachy?

Outreachy is a three-month internship program. It’s completely remote (both interns and mentors come from around the world). We pay the interns a $5,500 USD stipend for the three months, plus a $500 travel stipend to attend a conference or event related to their internship or free software.

The goal of the internship is to introduce people to free and open source software. Outreachy has projects that involve programming, documentation, graphic design, user experience, user advocacy, and data science.

Outreachy’s other goal is to support people from groups underrepresented in the technology industry. We expressly invite women (both cis and trans), trans men, and genderqueer people to apply. We also expressly invite applications from residents and nationals of the United States of any gender who are Black/African American, Hispanic/Latin@, Native American/American Indian, Alaska Native, Native Hawaiian, or Pacific Islander. Anyone who faces under-representation, systemic bias, or discrimination in the technology industry of their country is invited to apply.

What’s My Role?

I own Otter Tech LLC, which is a diversity and inclusion consulting company. It’s been my full-time job since July 2016. I work with clients (mostly in the technology or free software space) that want to improve their culture and better support people from groups underrepresented in tech. Outreachy is one of my clients.

I am one of five Outreachy organizers. Two of us (Marina Zhurakhinskaya and I) are heavily involved in running the internship application process. Karen Sandler is great at finding funding for us. The whole Outreachy organizers team (including Tony Sebro and Cindy Pallares-Quezada) makes important decisions about the direction of the program.

Outreachy also recently hired two part-time staff members. They’ve been helping Outreachy applicants during the application period, and then also helping Outreachy interns when the internship is running. We don’t have a good name for their role yet, but we’ve sort of settled on “Outreachy Helpers”

December 2018 Progress

The December 2018 to March 2018 internship round kicked off on December 4. Usually that’s downtime for me as an Outreachy organizer, because mentors and coordinators step up to interact with their interns. In the past, the only real interaction the Outreachy organizers had with interns was if their mentor indicated they were having issues (yikes!). This month was spent increasing the frequency and types of check-ins with interns and mentors.

Outreachy Chat Server

This round, we’re trying something new to have the Outreachy interns talk with Outreachy organizers and with each other. We’ve set up a private invitation-only Zulip chat server, and invited all the Outreachy organizers, interns, mentors, and coordinators. I’ve been doing a bit of community management, participating in discussions, and answering questions that Outreachy interns have as they start their internship. I also ran a text-based discussion and then a video chat for Outreachy interns to do a second week check-in.

I think the Outreachy Zulip chat has worked out well! I see interns connecting across different free software communities, and mentors from other communities helping different interns. Zulip has the concept of “streams” which are basically chat rooms. We have a couple of different streams, like a general chat channel and a channel for asking questions about Outreachy internship procedures. I’m fairly certain that I got more questions on the Zulip chat from interns than we ever got by using email and IRC.

Frequent Feedback

The other thing we’re doing this round is collecting feedback in a different way. In the past, we collected it at two points during the internship. The midpoint was at 6 weeks in and the final feedback was at 12 weeks in. However, this round, we’re collecting it at three points: initial feedback at 2 weeks in, midpoint feedback at 8 weeks in, and final feedback at 12 weeks.

Collecting feedback three times meant more overhead for evaluating feedback and sending the results to our fiscal sponsor, the Software Freedom Conservancy. I wrote code in December to allow the Outreachy internship website to collect feedback from mentors as to whether interns should be paid their initial stipend.

We’re also collecting different feedback this round. I’m collecting feedback from both interns and mentors, based on a suggestion from a former Outreachy intern. Interns and mentors are asked the same questions, like “How long does it take (you/your intern) to respond to questions or feedback?” and “How long does it take (your mentor/you) to respond to questions and feedback?” That way, I can compare people’s self-evaluations with what the other person involved in the internship thinks.

There’s also a freeform-text for interns to give feedback on how their mentor is doing. This is important, because many Outreachy mentors are new to mentoring. They may need to have some coaching to understand how they can be more supportive to their interns. While most of the interns are doing great, I can see that I’m going to need to nudge a couple of mentor and intern pairs in the right direction.

Interviews with Alums

I did video interviews with five Outreachy interns at the Mozilla All Hands in December 2019. I loved interviewing them, because it’s great to hear their personal stories. I’ll be using the footage to create videos to promote the Outreachy program.

I’ve created short-hand transcripts of two of the videos, but haven’t gotten to the other five. Transcripts help for a couple reasons. Most importantly, I can add closed captioning to the finished videos. I also have a searchable text database for when I need to find quotes about a particular topic. Seeing the text allows me to group similar experiences and create a cohesive narrative for the promotional video.

Ramping up for May 2019 Internships

The Outreachy December 2018 to March 2019 internships are just starting, but we’re already thinking of the next round. January is typically the time we start pinging communities to see if they want to be involved in mentoring interns during the February to March application period.

That means we need to have the website ready to handle both a currently running internship cohort, and a new internship round where mentors can submit projects. There’s some technical debt in the Outreachy website code that we need to address before we can list the next round’s internship dates.

The Outreachy website is designed to guide internship applicants through the application process. It’s built with a web framework tool called Django, which is written in Python. Django makes web development easier, because you can define Python classes that represent your data. Django then uses those classes to create a representation in the database. The part of Django that translates Python into database schema is called the ORM (Object Relational Mapper).

For example, the Outreachy website keeps track of internship rounds (the RoundPage class). Each internship round has dates and other information associated with it. For example, it has the date for when the application period starts and ends, and when the internship starts and end.

It makes sense to store internship rounds in a database, because all internship rounds have the same kinds of deadlines associated with them. You can do database queries to find particular rounds in the database. For example, the Django Python code to look up the latest round (based on when the interns start their internship) is RoundPage.objects.latest(‘internstarts’).

The work I’ve recently been doing is to deal with the fact that two internship rounds can be active at once. We’re about to open the next internship round for mentors to submit new projects. On February 18, the next application period will open. But the December 2018 round of internships will still be active until March 4.

The Outreachy website’s pages has to deal with displaying data from multiple rounds. For example, on the Outreachy organizers’ dashboard page, I need to be able to send out reminder emails about final mentor feedback for the December 2018 round, while still reviewing and approving new communities to participate in the May 2019 round. Outreachy mentors need to still be able to submit feedback for their current intern in the December 2018 round, while (potentially) submitting a new project for the May 2019 round.

It’s mostly a lot of refactoring and debugging Python code. I’m writing more Django unit tests to deal with corner cases. Sometimes it’s hard to debug when something fails in the unit test, but doesn’t fail in our local deployment copy. I’m fairly new to testing in Django, and I wrote my first test recently! I feel really silly for not starting on the tests sooner, but I’m slowly catching up to things!

What’s Next?

January 2019 is going to be spent contacting communities about participating in the May 2018 to August 2018 round. I have some video footage of Outreachy interns I interviewed at the Tapia conference and Mozilla All Hands, and I hope to put it into a promotional video to inspire people to become mentors. It’s a fun exercise that uses some of the video editing skills I have from making fanvideos.

I’ll also be at FOSDEM in February 2019. If you’re there, find me in either the Software Freedom Conservancy booth on Saturday, or the Community devroom on Sunday. I’ll also be helping out with the Copyleft Conference on Monday.

I’ll be giving a talk at FOSDEM on changing team culture to better support people with impostor syndrome. The goal is not to ask people with impostor syndrome to change, but instead to figure out how to change our culture so that we don’t create or trigger impostor syndrome. The talk is called “Supporting FOSS Community Members with Impostor Syndrome“. The talk will be from 9:10am to 9:40am on Sunday (the first talk slot).


Update on Sentiment Analysis of FOSS communities

One of my goals with my new open source project, FOSS Heartbeat, has been to measure the overall sentiment of communication in open source communities. Are the communities welcoming and friendly, hostile, or neutral? Does the bulk of positive or negative sentiment come from core contributors or outsiders? In order to make this analysis scale across multiple open source communities with years of logs, I needed to be able to train an algorithm to recognize the sentiment or tone of technical conversation.

How can machine learning recognize human language sentiment?

One of the projects I’ve been using is the Stanford CoreNLP library, an open source Natural Language Processing (NLP) project. The Stanford CoreNLP takes a set of training sentences (manually marked so that each word and each combined phrase has a sentiment) and it trains a neural network to recognize the sentiment.

The problem with any form of artificial intelligence is that the input into the machine is always biased in some way. For the Stanford CoreNLP, their default sentiment model was trained on movie reviews. That means, for example, that the default sentiment model thinks “Christian” is a very positive word, whereas in an open source project that’s probably someone’s name. The default sentiment model also consistently marks any sentence expressing a neutral technical opinion as having a negative tone. Most people leaving movie reviews either hate or love the movie, and people are unlikely to leave a neutral review analyzing the technical merits of the special effects. Thus, it makes sense that a sentiment model trained on movie reviews would classify technical opinions as negative.

Since the Stanford CoreNLP default sentiment model doesn’t work well on technical conversation, I’ve been creating a new set of sentiment training data that only uses sentences from open source projects. That means that I have to manually modify the sentiment of words and phrases in thousands of sentences that I feed into the new sentiment model. Yikes!

As of today, the Stanford CoreNLP default sentiment model has ~8,000 sentences in their training file. I currently have ~1,200 sentences. While my model isn’t as consistent as the Stanford CoreNLP, it is better at recognizing neutral and positive tone in technical sentences. If you’re interested in the technical details (e.g. specificity, recall, false positives and the like), you can take a look at the new sentiment model’s stats. This blog post will attempt to present the results without diving into guided machine learning jargon.

Default vs New Models On Positive Tone

Let’s take a look at an example of a positive code review experience. The left column is from the default sentiment model in Stanford CoreNLP, which was trained on movie reviews. The right column is from the new sentiment model I’ve been training. The colors of the sentence encode what the two models think the overall tone of the sentence is:

  • Very positive
  • Positive
  • Neutral
  • Negative
  • Very negative

Hey @1Niels 🙂 is there a particular reason for calling it Emoji Code?

I think the earlier guide called it emoji name.

A few examples here would help, as well as explaining that the pop-up menu shows the first five emojis whose names contain the letters typed.

(I’m sure you have a better way of explaining this than me :-).

@arpith I called them Emoji code because that’s what they’re called on Slack’s emoji guide and more commonly only other websites as well.

I think I will probably change the section name from Emoji Code to Using emoji codes and I’ll include your suggestion in the last step.

Thanks for the feedback!

Hey @1Niels 🙂 is there a particular reason for calling it Emoji Code?

I think the earlier guide called it emoji name.

A few examples here would help, as well as explaining that the pop-up menu shows the first five emojis whose names contain the letters typed.

(I’m sure you have a better way of explaining this than me :-).

@arpith I called them Emoji code because that’s what they’re called on Slack’s emoji guide and more commonly only other websites as well.

I think I will probably change the section name from Emoji Code to Using emoji codes and I’ll include your suggestion in the last step.

Thanks for the feedback!

Default vs New Models On Positive Tone

For the default model trained on movie reviews, it rated 4 out of 7 of the sentences as negative and 1 out of 7 sentences as positive. As you can see, the default sentiment model that was trained on movie reviews tends to classify neutral technical talk as having a negative tone, including sentences like “I called them Emoji code because that’s what they’re called on Slack’s emoji guide and more commonly only other websites as well.” It did recognize the sentence “Thanks for the feedback!” as positive, which is good.

For the new model trained on comments from open source projects, it rated 1 sentence as negative, 2 as positive, and 1 as very positive. Most of the positive tone of this example comes from the use of smiley faces, which I’ve been careful to train the new model to recognize. Additionally, I’ve been teaching it that exclamation points ending a sentence that is overall positive shift the tone to very positive. I’m pleased to see it pick up on those subtleties.

Default vs New Models On Neutral Tone

Let’s have a look at a neutral tone code review example. Again, the sentence sentiment color key is:

  • Very positive
  • Positive
  • Neutral
  • Negative
  • Very negative

This seems to check resolvers nested up to a fixed level, rather than checking resolvers and namespaces nested to an arbitrary depth.

I think a inline-code is more appropriate here, something like “URL namespace {} is not unique, you may not be able to reverse all URLs in this namespace”.

Errors prevent management commands from running, which is a bit severe for this case.

One of these should have an explicit instance namespace other than inline-code, otherwise the nested namespaces are not unique.

Please document the check in inline-code.

There’s a list of URL system checks at the end.

This seems to check resolvers nested up to a fixed level, rather than checking resolvers and namespaces nested to an arbitrary depth.

I think a inline-code is more appropriate here, something like “URL namespace {} is not unique, you may not be able to reverse all URLs in this namespace”.

Errors prevent management commands from running, which is a bit severe for this case.

One of these should have an explicit instance namespace other than inline-code, otherwise the nested namespaces are not unique.

Please document the check in inline-code.

There’s a list of URL system checks at the end.

Default vs New Models On Neutral Tone

Again, the default sentiment model trained on movie reviews classifies neutral review as negative, ranking 5 out of 6 sentences as negative.

The new model trained on open source communication is a bit mixed on this example, marking 1 sentence as positive and 1 negative, out of 6 sentences. Still, 4 out of 6 sentences were correctly marked as neutral, which is pretty good, given the new model has a training set that is 8 times smaller than the movie review set.

Default vs New Models On Negative Tone

Let’s take a look at a negative example. Please note that this is not a community that I am involved in, and I don’t know anyone from that community. I found this particular example because I searched for “code of conduct”. Note that the behavior displayed on the thread caused the initial contributor to offer to abandon their pull request. A project outsider stated they would recommend their employer not use the project because of the behavior. Another project member came along to ask for people to be more friendly. So quite a number of people thought this behavior was problematic.

Again, the sentiment color code is:

  • Very positive
  • Positive
  • Neutral
  • Negative
  • Very negative

Dude, you must be kidding everyone.

What dawned on you – that for a project to be successful and useful it needs confirmed userbase – was crystal clear to others years ago.

Your “hard working” is little comparing to what other people have been doing for years.

Get humbler, Mr. Arrogant.

If you find this project great, figure out that it is so because other people worked on it before.

Learn what they did and how.

But first learn Python, as pointed above.

Then keep working hard.

And make sure the project stays great after you applied your hands to it.

Dude, you must be kidding everyone.

What dawned on you – that for a project to be successful and useful it needs confirmed userbase – was crystal clear to others years ago.

Your “hard working” is little comparing to what other people have been doing for years.

Get humbler, Mr. Arrogant.

If you find this project great, figure out that it is so because other people worked on it before.

Learn what they did and how.

But first learn Python, as pointed above.

Then keep working hard.

And make sure the project stays great after you applied your hands to it.

Default vs New Models On Negative Tone

For the default model trained on movie reviews, it classifies 4 out of 9 sentences as negative and 2 as positive. The new model classifies 2 out of 9 sentences as negative and 2 as positive. In short, it needs more work.

It’s unsurprising that the new model doesn’t currently recognize negative sentiment very well right now, since I’ve been focusing on making sure it can recognize positive sentiment and neutral talk. The training set currently has 110 negative sentences out of 1205 sentences total. I simply need more negative examples, and they’re hard to find because many subtle personal attacks, insults, and slights don’t use curse words. If you look at the example above, there’s no good search terms, aside from the word arrogant, even though the sentences are still put-downs that create an us-vs-them mentality. Despite not using slurs or curse words, many people found the thread problematic.

The best way I’ve settled on to find negative sentiment examples is to look for “communication meta words” or people talking about communication style. My current list of search terms includes words like “friendlier”, “flippant”, “abrasive”, and similar. Some search words like “aggressive” yield too many false positives, because people talk about things like “aggressive optimization”. Once I’ve found a thread that contains those words, I’ll read through it and find the comments that caused the people to ask for a different communication style. Of course, this only works for communities that want to be welcoming. For other communities, searching for the word “attitude” seems to yield useful examples.

Still, it’s a lot of manual labor to identify problematic threads and fish out the negative sentences that are in those threads. I’ll be continuing to make progress on improving the model to recognize negative sentiment, but it would help if people could post links to negative sentiment examples on the FOSS Heartbeat github issue or drop me an email.

Visualizing Sentiment

Although the sentiment model isn’t perfect, I’ve added visualization for the sentiment of several communities on FOSS Heartbeat, including 24pullrequests, Dreamwidth, systemd, elm, fsharp, and opal.

The x-axis is the date. I used the number of neutral comments in an issue or pull request as the y-axis coordinate, with the error bars indicating the number of positive and negative comments. If the comment had two times the number of negative comments as positive comments, it was marked as a negative thread. If the comment had two times the number of positive comments than negative comments, it was marked as positive. If neither sentiment won, and more than 80% of the comments were neutral, it was marked as neutral. Otherwise the issue or pull request was marked as mixed sentiment.

Here’s an example:

24pullrequests-sentiment

The sentiment graph is from the 24pullrequests repository. It’s a ruby website that encourages programmers to gift code to open source projects during the 24 days in December before Christmas. One of the open source projects you can contribute to is the 24 pull requests site itself (isn’t that meta!). During the year, you’ll see the site admins filing help-wanted enhancements to update the software that runs the website or tweak a small feature. They’re usually closed within a day without a whole lot of back and forth between the main contributors. The mid-year contributions show up as the neutral, low-comment dots throughout the year. When the 24 pull request site admins do receive a gift of code to the website by a new contributor as part of the 24 pull requests period, they’re quite thankful, which you can see reflected in the many positive comments around December and January.

Another interesting example to look at is negative sentiment in the opal community:

opal-negative-sentiment

That large spike with 1207 neutral comments, 197 positive comments, and 441 negative comments is the opal community issue to add a code of conduct. Being able to quickly see which threads are turning into flamewars would be helpful to community managers and maintainers who have been ignoring the issue tracker to get some coding done. Once the sentiment model is better trained, I would love to analyze whether communities become more positive or more neutral after a Code of Conduct is put in place. Tying that data to whether more or less newcomers participate after a Code of Conduct is in place may be interesting as well.

There are a lot of real-world problems that sentiment analysis, participation data, and a bit of psychology could help us identify. One common social problem is burnout, which is characterized by an increased workload (stages 1 & 2), working at odd hours (stage 3), and an increase in negative sentiment (stage 6). We have participation data, comment timestamps, and sentiment for those comments, so we would only need some examples of burnout to identify the pattern. By being aware of the burnout stages of our collaborators, we could intervene early to help them avoid a spiral into depression.

A more corporate focused interest might be to identify issues where their key customers express frustration and anger, and focus their developers on fixing the squeaky wheel. If FOSS Heartbeat were extended to analyze comments on mailing lists, slack, discourse, or mattersmost, companies could get a general idea of the sentiment of customers after a new software release. Companies can also use the participation and data about who is merging code to figure out which projects or parts of their code are not being well-maintained, and assign additional help, as the exercism community did.

Another topic of interest to communities hoping to grow their developer base would be identifying the key factors that cause newcomers to become more active contributors to a project. Is it a positive welcome? A mentor suggesting a newcomer tackle a medium-sized issue by tagging them? Does adding documentation about a particularly confusing area cause more newcomers to submit pull requests to that area of code? Does code review from a particularly friendly person cause newcomers to want to come back? Or maybe code review lag causes them to drop off?

These are the kinds of people-centric community questions I would love to answer by using FOSS Heartbeat. I would like to thank Mozilla for sponsoring the project for the last three months. If you have additional questions you’d love to see FOSS Heartbeat answer, I’m available for contract work through Otter Tech. If you’re thankful about the work I’ve put in so far, you can support me through my patreon.

What open source community question would you like to see FOSS Heartbeat tackle? Feel free to leave a comment.