GBI: Teaching Perceptions Unveiled: Analyzing Gender Bias and Predictors in Professor Ratings

Project Overview

This project investigates gender bias in professor ratings and identifies key predictors of teaching effectiveness using data from RateMyProfessor.com (RMP). I analyzed disparities in ratings, examined descriptive tags, and built predictive models to uncover patterns in evaluations. The findings aim to improve fairness in academia and provide actionable insights for educators and institutions.

Data Collection

The dataset was created using the following steps:

1. Web Scraping:

   • Scraped professor profiles, ratings, tags, and institutional data from RMP using Python libraries like BeautifulSoup and Selenium.

2. Data Collation:

   • Aggregated individual ratings into average scores (e.g., quality, difficulty) and cleaned duplicates.

3. Anonymization:

   • Removed identifying information such as professor names to preserve privacy.

4. Validation:

   • Cross-checked with a manually collected sample, achieving a 98% match rate for accuracy.

Preprocessing Challenges and Solutions

Challenges Encountered

1. Missing Data:

   • Several records were incomplete, with missing values for critical fields like average ratings and difficulty levels.

   • Problem: Missing values reduce the dataset’s analytical integrity and may lead to biased conclusions.

2. Extreme Ratings:

   • Some professors had ratings based on only one or two students, leading to extreme averages of 1 or 5.

   • Problem: Such low-sample averages are unreliable and can skew statistical analysis.

3. False Positives in Statistical Analysis:

   • The large dataset provided sufficient statistical power, increasing the risk of false positives when conducting multiple hypothesis tests.

   • Problem: Without proper corrections, spurious results could compromise the validity of findings.

4. Raw Tag Counts:

   • Tags like "Tough Grader" were recorded as raw counts, biased by the number of ratings a professor received.

   • Problem: Professors with more ratings appeared to have higher tag counts, distorting meaningful comparisons.

5. Ambiguous Gender Entries:

   • Some records had unclear or conflicting gender classifications.

   • Problem: This ambiguity could mislead gender-based analyses, reducing credibility.

How Challenges Were Addressed

1. Handling Missing Data:

   • Do: Rows with missing average ratings or difficulty values were dropped to ensure data integrity.

   • Why: Excluding incomplete records guarantees that analyses rely on complete and valid information.

   • Find: This step reduced noise in the dataset, allowing for more reliable statistical results.

2. Filtering Extreme Ratings:

   • Do: Professors with fewer than five ratings were excluded.

   • Why: A minimum threshold prevents skewed averages caused by a small number of ratings, fostering more stable insights.

   • Find: After filtering, the average rating distribution remained centered around ~3.5, reflecting typical evaluations.

3. Mitigating False Positives:

   • Do: Applied a stricter significance threshold (( \alpha = 0.005 )) for all statistical tests and employed Bonferroni correction for multiple comparisons.

   • Why: These adjustments reduce the likelihood of false positives, ensuring robust and credible findings.

   • Find: The adjusted threshold identified significant patterns while avoiding overinterpretation.

4. Normalizing Tags:

   • Do: Converted raw tag counts to proportions (tag count ÷ total ratings).

   • Why: Normalizing tags ensures fair comparisons, regardless of the number of ratings a professor received.

   • Find: This step revealed meaningful differences in tag usage across genders, with certain characteristics like "Tough Grader" appearing more frequently.

5. Resolving Ambiguous Gender Entries:

   • Do: Created binary columns (gender00, gender11) and assigned ambiguous cases a value of 0 for both Male and Female.

   • Why: Explicitly handling ambiguous entries prevents misclassification and strengthens gender-based analyses.

   • Find: Gender data became cleaner and more interpretable, supporting accurate statistical tests.

6. Seeding the Random Number Generator:

   • Do: Seeded the RNG with a team member's unique identifier (N-number).

   • Why: Ensures reproducibility of results, allowing consistent splits in train-test datasets and bootstrapping processes.

   • Find: Reproducibility helped maintain computational integrity throughout the project.
     
     

Objectives

With this dataset in hand, the project aims to answer the following questions:

1. Gender Bias in Ratings:

   • Activists have asserted that there is a strong gender bias in student evaluations of professors, with male professors enjoying a boost in ratings. However, skeptics have criticized the quality of such research, citing small sample sizes, failure to control for confounders, and potential p-hacking. This project examines whether there is evidence of a pro-male gender bias in the dataset.

   • Hint: A significance test is probably required.

2. Variance in Ratings:

   • Is there a gender difference in the spread (variance/dispersion) of the ratings distribution? Statistical significance of observed differences in variance will be evaluated.

3. Effect Size Estimation:

   • What is the likely size of both of these effects (gender bias in average rating, gender bias in spread of average rating)? This will be estimated using a 95% confidence interval.

4. Gender Differences in Tags:

   • Are there gender differences in the tags awarded by students? Each of the 20 tags will be tested for potential gender differences, identifying the most and least gendered tags based on statistical significance (lowest and highest p-values, respectively).

5. Gender Differences in Difficulty Ratings:

   • Is there a gender difference in terms of average difficulty? Statistical testing will be used to determine significance.

6. Quantifying Effect Size in Difficulty:

   • What is the likely size of the gender difference in difficulty ratings at a 95% confidence level?

7. Predicting Average Ratings:

   • Build a regression model to predict average ratings using all numerical predictors from the rmpCapstoneNum.csv file. The model should include metrics such as ( R^2 ) and RMSE, and address collinearity concerns to identify the most significant predictors.

8. Predicting Ratings Using Tags:

   • Build a regression model to predict average ratings using all tags from the rmpCapstoneTags.csv file. Compare this model's performance (e.g., ( R^2 ), RMSE) with the numerical model and identify which tags are most strongly predictive of ratings.

9. Predicting Difficulty Using Tags:

   • Build a regression model to predict average difficulty using all tags from the rmpCapstoneTags.csv file. Include metrics like ( R^2 ) and RMSE, and identify the most significant predictors while addressing collinearity.

10. Predicting Pepper Badges:

    • Build a classification model to predict whether a professor receives a "pepper" badge using all available factors (tags and numerical). Evaluate model quality with metrics such as AU(RO)C and address class imbalance concerns.