Abeed Sarker, Maksim Belousov, Jasper Friedrichs, Kai Hakala, Svetlana Kiritchenko, Farrokh Mehryary, Sifei Han, Tung Tran, Anthony Rios, Ramakanth Kavuluru, Berry de Bruijn, Filip Ginter, Debanjan Mahata, Saif M Mohammad, Goran Nenadic, and Graciela Gonzalez-Hernandez.

Journal of the American Medical Informatics Association (JAMIA), 2018.

Abstract

Objective

We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data.

Materials and Methods

We organized 3 independent subtasks: automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks.

Results

Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F1-score) for subtask-1, 0.693 (micro-averaged F1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems.

Discussion

Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1).

Conclusions

Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).

[Link]

@article{doi:10.1093/jamia/ocy114,
 author = {Sarker, Abeed and Belousov, Maksim and Friedrichs, Jasper and Hakala, Kai and Kiritchenko, Svetlana and Mehryary, Farrokh and Han, Sifei and Tran, Tung and Rios, Anthony and Kavuluru, Ramakanth and de Bruijn, Berry and Ginter, Filip and Mahata, Debanjan and Mohammad, Saif M and Nenadic, Goran and Gonzalez-Hernandez, Graciela},
 title = {Data and systems for medication-related text classification and concept normalization from Twitter: insights from the Social Media Mining for Health (SMM4H)-2017 shared task},
 journal = {Journal of the American Medical Informatics Association},
 pages = {ocy114},
 year = {2018},
 doi = {10.1093/jamia/ocy114},
 URL = {http://dx.doi.org/10.1093/jamia/ocy114},
 eprint = {/oup/backfile/content_public/journal/jamia/pap/10.1093_jamia_ocy114/2/ocy114.pdf}
}