Research procedure
Classify online messages
- Five coders annotate 10,500 messages
- Check agreement; train a random forest
- Cross-validate; classify the remaining corpus
Illustrative summary of the source procedure. Agreement was checked on 500 shared messages. Five-fold cross-validation gave F1 of at least 0.8 for business content, sentiment and audience scope in a corpus of over 140,000 messages.
See source →How
When to use
Organizations need to categorize large volumes of unstructured text messages from online platforms. Manual coding by human researchers becomes impossible when message counts scale into the hundreds of thousands. Automated machine-learning classifiers trained on a representative sample allow researchers to analyze semantic patterns across massive text datasets.
Classify unstructured message data into semantic categories using human annotation and supervised machine learning.
- Five human coders manually annotate a random sample of 10,500 messages from the platform to label business content, sentiment, and audience scope.
- Researchers calculate inter-coder agreement across a shared subset of 500 messages using the kappa statistic to verify coding consistency.
- Analysts train a random forest machine-learning algorithm on the annotated training data using aggregated decision tree models.
- Analysts evaluate model performance via 5-fold cross-validation to achieve an F1 score of at least 0.8 across target categories.
- The algorithm classifies the remaining corpus of over 140,000 messages to assign category weights for every sentence.
Why it works
Training on a reliably annotated human sample allows automated models to achieve high accuracy when classifying unseen communication.
Example
Data scientists annotated 10,500 messages sent by African entrepreneurs to train a random forest model that classified over 140,000 sentences exchanged across the Adansonia platform, identifying business focus, sentiment, and audience scope with high cross-validation reliability.
About the study
Participants
140,000+ messages sent by 4,958 entrepreneurs across 49 African countries; 10,500 messages manually annotated by 5 human coders.
Study Design
Supervised random forest classification with inter-coder agreement measured via Cohen's kappa (\\(\kappa = 0.67\\)) and model reliability evaluated through 5-fold cross-validation.
Results
The random forest classifier achieved high predictive reliability across the dataset with \\(F_1 \ge 0.8\\) for business content, sentiment, and audience scope, enabling full semantic labeling across all 140,000+ messages.