首页 > 其他 > 详细

fasttext的使用,预料格式,调用方法

时间:2020-02-20 01:20:43      阅读:410      评论:0      收藏:0      [点我收藏+]

数据格式:分词后的句子+\t__label__+标签

技术分享图片

fasttext_model.py

from fasttext import FastText
import numpy as np

def get_data_path(by_word=True,train=True):
    if by_word:
        return "./classify/data_by_word_train.txt" if train else "./classify/data_by_word_test.txt"
    else:
        return "./classify/data_train.txt" if train else "./classify/data_test.txt"

def prepar_model():
    data_path = get_data_path(by_word=True,train=True)
    model = FastText.train_supervised(data_path,dim=100,epoch=20,wordNgrams=2)
    model.save_model("./fasttext_model/classify_by_word_100_20_2.model")

def ceshi_model():
    model = FastText.load_model("./fasttext_model/classify_by_word_100_20_2.model")
    test_data_path = get_data_path(by_word=True, train=False)

    sentences = []
    labels = []
    for line in open(test_data_path,encoding="utf-8").readlines():
        line = line.strip()
        temp_ret = line.split("\t")
        if len(temp_ret)==2:
            sentences.append(temp_ret[0])
            labels.append(temp_ret[1])

    ret = model.predict(sentences)[0]
    ret = [i[0] for i in ret]
    acc = np.mean([1 if labels[i] == ret[i] else 0 for i in range(len(labels))])
    print(acc)

if __name__ == ‘__main__‘:
    prepar_model()
    ceshi_model()

  

 

fasttext的使用,预料格式,调用方法

原文:https://www.cnblogs.com/LiuXinyu12378/p/12334002.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!