word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。_word2vec unicodedecodeerror: 'utf-8' codec can't d

原文链接：https://blog.csdn.net/W_Honor/article/details/105037033?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param

NLP领域最出名的python库之一就是gensim，该库包含了常见的word2vec模型，而我们在使用这些模型进行embedding的时候可会出现如下的编码问题：

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position

个错误在提示加载模型时出现了编码错误，即在某一个位置的字节是无法编码的，并会给出相应的的位置。
这个问题在gensim官方github中也有反馈，并且作者统一给出了解决方法：

Answer: The strings (words) stored in your model are not valid utf8. By default, gensim decodes the words using the strict encoding settings, which results in the above exception whenever an invalid utf8 sequence is encountered.
The fix is on your side and it is to either:
a) Store your model using a program that understands unicode and utf8 (such as gensim). Some C and Java word2vec tools are known to truncate the strings at byte boundaries, which can result in cutting a multi-byte utf8 character in half, making it non-valid utf8, leading to this error.
b) Set the unicode_errors flag when running load_word2vec_model, e.g. load_word2vec_model(…, unicode_errors=‘ignore’). Note that this silences the error, but the utf8 problem is still there – invalid utf8 characters will just be ignored in this case.

第一种方法告诉了使用者可以用C和JAVA的相关工具去处文本中的无效编码字符；
第二种方法建议在load_word2vec_model函数中设置参数unicode_errors=‘ignore’，可以暂时忽略这个错误。

def get_word_vectors(model_path, dim, words):
    wordVec = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True, unicode_errors='ignore')
    wordEmbedding = []
    for word in words:
        try:
            if "<PAD>" == word:
                wordEmbedding.append(np.zeros(dim))
            else:
                vector = wordVec.wv[word]
                wordEmbedding.append(vector)
        except:
            print(word + "不存在于词向量中")
            wordEmbedding.append(np.random.randn(dim))
 
    return np.array(wordEmbedding)