anti-cursing
"anti-cursing" is a python package that detects and switches negative or any kind of cursing word from sentences or comments whatever
You just install the package the way you install any other package and then you can use it in your code.
The whole thing is gonna be updated soon.
So this is the very first idea
But you can find my package in pypi(https://pypi.org/project/anti-cursing/0.0.1/)
ππ»Plz bare with the program to install model's weight and bias from huggingface at the first time you use the package.
Concept
There are often situations where you have to code something, detect a forbidden word, and change it to another word. Hardcoding all parts is very inconvenient, and in the Python ecosystem, there are many packages to address. One of them is "anti-cursing".
The package, which operates exclusively for Korean, does not simply change the banned word by setting it up, but detects and replaces the banned word by learning a deep learning model.
Therefore, it is easy to cope with new malicious words as long as they are learned. For this purpose, semi-supervied learning through pseudo labeling is used.
Additionally, instead of changing malicious words to special characters such as --- or ***, you can convert them into emojis to make them more natural.
Contents
- Installation
- Usage
- Model comparison
- Dataset
- Used API
- License
- Working Example
- References
- Project Status
- Future Work
Installation
You can install the package using pip:
pip install anti-cursing
it doesn't work yet, but it will soon!!π¨π»βπ»
Usage
from anti_cursing.utils import antiCursing
antiCursing.anti_cur("λλ λκ° μ’μ§λ§, λλ λ무 κ°μλΌμΌ")
λλ λκ° μ’μ§λ§, λλ λ무 πΌπ»μΌ
Model-comparison
Classification | KcElectra | KoBERT | RoBERTa-base | RoBERTa-large |
---|---|---|---|---|
Validation Accuracy | 0.88680 | 0.85721 | 0.83421 | 0.86994 |
Validation Loss | 1.00431 | 1.23237 | 1.30012 | 1.16179 |
Training Loss | 0.09908 | 0.03761 | 0.0039 | 0.06255 |
Epoch | 10 | 40 | 20 | 20 |
Batch-size | 8 | 32 | 16 | 32 |
transformers | beomi/KcELECTRA-base | skt/kobert-base-v1 | xlm-roberta-base | klue/roberta-large |
Dataset
-
- https://github.com/smilegate-ai/korean_unsmile_dataset
- Korean Sentiment Analysis
- paper
-
- https://news.naver.com
- Non-labeled Data for Test Dataset
-
π Emoji unicode crawling for encoding
Used-api
Google translator
- https://cloud.google.com/translate/docs (API DOCS)
License
This repository is licensed under the MIT license. See LICENSE for details.
Click here to see the License information --> License
Working-example
---- some video is gonna be placed here ----
References
Sentiment Analysis Based on Deep Learning : A Comparative Study
- Nhan Cach Dang, Maria N. Moreno-Garcia, Fernando De la Prieta. 2006. Sentiment Analysis Based on Deep Learning : A Comparative Study. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 1β8, Prague, Czech Republic. Association for Computational Linguistics.
Attention is all you need
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000β6010.
BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 4171β4186.
Electra : Pre-training Text Encoders as Discriminators Rather Than Generators
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 4171β4186.
BIDAF : Bidirectional Attention Flow for Machine Comprehension
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi. 2016. Bidirectional Attention Flow for Machine Comprehension. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2129β2139.
Effect of Negation in Sentences on Sentiment Analysis and Polarity Detection
- Partha Mukherjeea, Saptarshi Ghoshb, and Saptarshi Ghoshc. 2018. Effect of Negation in Sentences on Sentiment Analysis and Polarity Detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2129β2139.
KOAS : Korean Text Offensiveness Analysis System
- Seonghwan Kim, Seongwon Lee, and Seungwon Do. 2019. KOAS: Korean Text Offensiveness Analysis System. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1β11.
Korean Unsmile Dataset
- Seonghwan Kim, Seongwon Lee, and Seungwon Do. 2019. Korean Unsmile Dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1β11.
Project-status
Future-work
update soon plz bare with me ππ»
KOREAN FROM HERE / μ¬κΈ°λΆν΄ νκ΅μ΄ μ€λͺ μ λλ€.
anti-cursing
**"anti-cursing"**μ λ¬Έμ₯μ΄λ λκΈμμ λΆμ μ μ΄κ±°λ λͺ¨λ μ’
λ₯μ μμ€μ κ°μ§νκ³ μ ννλ νμ΄μ¬ ν¨ν€μ§μ
λλ€
λ€λ₯Έ ν¨ν€μ§λ₯Ό μ€μΉνλ λ°©μκ³Ό λμΌνκ² ν¨ν€μ§λ₯Ό μ€μΉν λ€μ μ½λμμ μ¬μ©ν μ μμ΅λλ€.
μμ§ μμ΄λμ΄ κ΅¬μ λ¨κ³μ΄κΈ° λλ¬Έμ μ무κ²λ μλνμ§ μμ§λ§ 곧 μλνλλ‘ μ λ°μ΄νΈν μμ μ λλ€.
Pypi(https://pypi.org/project/anti-cursing/0.0.1/)μ ν¨ν€μ§λ₯΄ μ λ‘λνμ΅λλ€. νμΈνμ μ μμ΅λλ€.
ππ»ν¨ν€μ§λ₯Ό μ²μ μ€μΉνμκ³ μ¬μ©νμ€ λ λ₯λ¬λ λͺ¨λΈμ λΆλ¬μ€κΈ° μν΄ huggingfaceμμ parsingμ μλν©λλ€. μ²μμλ§ ν΄λΉ μμ μ΄ νμνλ μκ°μ΄ μ‘°κΈ κ±Έλ¦Όκ³Ό μ©λμ μ°¨μ§ν¨μ κ³ λ €ν΄μ£ΌμΈμ
Concept
무μΈκ° μ½λ©μ νλ©°, κΈμ§ λ¨μ΄λ₯Ό κ°μ§νκ³ κ·Έκ²μ λ€λ₯Έ λ¨μ΄λ‘ λ°κΏμΌν μν©μ΄ μ’ μ’ μκΉλλ€. λͺ¨λ λΆλΆμ νλμ½λ©νλ κ²μ΄ λ§€μ° λΆνΈνλ©°, νμ΄μ¬ μνκ³μμλ μ΄λ₯Ό ν΄κ²°νκΈ° μν λ§μ ν¨ν€μ§κ° μμ΅λλ€. κ·Έ μ€ νλκ° **"anti-cursing"**μ λλ€.
νκ΅μ΄ μ μ©μΌλ‘ λμνλ ν΄λΉ ν¨ν€μ§λ λ¨μν κΈμ§ λ¨μ΄λ₯Ό κΈ°μ‘΄μ μ€μ νμ¬ λ°κΎΈλ κ²μ΄ μλ, λ₯λ¬λ λͺ¨λΈμ νμ΅νμ¬ κΈμ§ λ¨μ΄λ₯Ό κ°μ§νκ³ λ°κΏλλ€. λ°λΌμ μλ‘κ² μκΈ°λ μ μ± λ¨μ΄μ λν΄μλ νμ΅λ§ μ΄λ£¨μ΄μ§λ€λ©΄ μ½κ² λμ²ν μ μμ΅λλ€. μ΄λ₯Ό μν΄ pseudo labelingμ ν΅ν semi-supervied learningμ μ¬μ©ν©λλ€.
μΆκ°λ‘ μ μ±λ¨μ΄λ₯Ό ---λ ***κ°μ νΉμλ¬Έμλ‘ λ³κ²½νλ κ²μ΄ μλ, μ΄λͺ¨μ§λ‘ λ³ννμ¬ λμ± μμ°μ€λ½κ² λ°κΏ μ μμ΅λλ€.
λͺ©μ°¨
- μ€μΉ
- μ¬μ©λ²
- λͺ¨λΈ μ±λ₯ λΉκ΅
- λ°μ΄ν°μ
- μ¬μ© API
- License
- μλ μμ
- μ°Έκ³ λ¬Έν
- μ§νμν©
- λ°μ
μ€μΉ
pipλ₯Ό μ¬μ©νμ¬ ν¨ν€μ§λ₯Ό μ€μΉν μ μμ΅λλ€.
pip install anti-cursing
μμ§ μ무κ²λ μλνμ§ μμ§λ§, 곧 μλνλλ‘ μ λ°μ΄νΈν μμ μ λλ€π¨π»βπ».
μ¬μ©λ²
from anti_cursing.utils import antiCursing
antiCursing.anti_cur("λλ λκ° μ’μ§λ§, λλ λ무 κ°μλΌμΌ")
λλ λκ° μ’μ§λ§, λλ λ무 πΌπ»μΌ
λͺ¨λΈ μ±λ₯ λΉκ΅
Classification | KcElectra | KoBERT | RoBERTa-base | RoBERTa-large |
---|---|---|---|---|
Validation Accuracy | 0.88680 | 0.85721 | 0.83421 | 0.86994 |
Validation Loss | 1.00431 | 1.23237 | 1.30012 | 1.16179 |
Training Loss | 0.09908 | 0.03761 | 0.0039 | 0.06255 |
Epoch | 10 | 40 | 20 | 20 |
Batch-size | 8 | 32 | 16 | 32 |
transformers | beomi/KcELECTRA-base | skt/kobert-base-v1 | xlm-roberta-base | klue/roberta-large |
λ°μ΄ν°μ
-
- https://github.com/smilegate-ai/korean_unsmile_dataset
- νκ΅μ΄ κ°μ λΆλ₯ λ°μ΄ν°μ
- paper
-
- https://news.naver.com
- ν μ€νΈλ₯Ό μν λ°μ΄ν°μ
-
π μ΄λͺ¨μ§ μ λμ½λ λ°μ΄ν°μ
μ¬μ© API
Google translator
- https://cloud.google.com/translate/docs (API λ¬Έμ)
License
μ΄ νλ‘μ νΈλ MIT λΌμ΄μΌμ€λ₯Ό λ°λ¦ λλ€. μμΈν λ΄μ©μ LICENSE νμΌμ μ°Έκ³ ν΄μ£ΌμΈμ.
λΌμ΄μΌμ€ μ 보 --> License
μλ μμ
---- μλ μμκ° μΆκ°λ μμ μ λλ€ ----
μ°Έκ³ λ¬Έν
Sentiment Analysis Based on Deep Learning : A Comparative Study
- Nhan Cach Dang, Maria N. Moreno-Garcia, Fernando De la Prieta. 2006. Sentiment Analysis Based on Deep Learning : A Comparative Study. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 1β8, Prague, Czech Republic. Association for Computational Linguistics.
Attention is all you need
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000β6010.
BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 4171β4186.
Electra : Pre-training Text Encoders as Discriminators Rather Than Generators
- Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 4171β4186.
BIDAF : Bidirectional Attention Flow for Machine Comprehension
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi. 2016. Bidirectional Attention Flow for Machine Comprehension. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2129β2139.
Effect of Negation in Sentences on Sentiment Analysis and Polarity Detection
- Partha Mukherjeea, Saptarshi Ghoshb, and Saptarshi Ghoshc. 2018. Effect of Negation in Sentences on Sentiment Analysis and Polarity Detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2129β2139.
KOAS : Korean Text Offensiveness Analysis System
- Seonghwan Kim, Seongwon Lee, and Seungwon Do. 2019. KOAS: Korean Text Offensiveness Analysis System. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1β11.
Korean Unsmile Dataset
- Seonghwan Kim, Seongwon Lee, and Seungwon Do. 2019. Korean Unsmile Dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1β11.
μ§νμν©
λ°μ
μμΌλ‘ μΆκ°λ μμ μ λλ€ μ μλ§ κΈ°λ€λ €μ£ΌμΈμππ»