嵌入时出错：无法将字符串转换为浮点数：'ng'

Question

我正在使用 GloVe 方法处理预训练的词向量。数据包含维基百科数据上的向量。在嵌入数据时，我收到错误，指出无法将字符串转换为浮点数：'ng'

我尝试浏览数据，但在那里我找不到符号“ng”

# load embedding as a dict

def load_embedding(filename):

    # load embedding into memory, skip first line

    file = open(filename,'r', errors = 'ignore')

    # create a map of words to vectors

    embedding = dict()

    for line in file:

        parts = line.split()

        # key is string word, value is numpy array for vector

        embedding[parts[0]] = np.array(parts[1:], dtype='float32')

    file.close()

    return embedding

这是错误报告。请进一步指导我。

runfile('C:/Users/AKSHAY/Desktop/NLP/Pre-trained GloVe.py', wdir='C:/Users/AKSHAY/Desktop/NLP')

C:\Users\AKSHAY\AppData\Local\conda\conda\envs\py355\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.

  from ._conv import register_converters as _register_converters

Using TensorFlow backend.

Traceback (most recent call last):

  File "<ipython-input-1-d91aa5ebf9f8>", line 1, in <module>

    runfile('C:/Users/AKSHAY/Desktop/NLP/Pre-trained GloVe.py', wdir='C:/Users/AKSHAY/Desktop/NLP')

  File "C:\Users\AKSHAY\AppData\Local\conda\conda\envs\py355\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile

    execfile(filename, namespace)

  File "C:\Users\AKSHAY\AppData\Local\conda\conda\envs\py355\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile

    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/AKSHAY/Desktop/NLP/Pre-trained GloVe.py", line 123, in <module>

    raw_embedding = load_embedding('glove.6B.50d.txt')

  File "C:/Users/AKSHAY/Desktop/NLP/Pre-trained GloVe.py", line 67, in load_embedding

    embedding[parts[0]] = np.array(parts[1:], dtype='float32')

ValueError: could not convert string to float: 'ng'

Answer 1

ValueError: 无法将字符串转换为浮点数：'ng'为了解决上述问题，在函数中添加encoding='utf8'如下：file&nbsp;=&nbsp;open(filename,'r',&nbsp;errors&nbsp;=&nbsp;'ignore',&nbsp;encoding='utf8')

Answer 2

这似乎工作正常：embedding_model = {}f = open(r'dataset/glove.840B.300d.txt', encoding="utf8", "r")for line in f:&nbsp; &nbsp; values = line.split()&nbsp; &nbsp; word = ''.join(values[:-300])&nbsp; &nbsp; coefs = np.asarray(values[-300:], dtype='float32')&nbsp; &nbsp; embedding_model[word] = coefsf.close()

Answer 3

看起来 'ng' 是您文件中的一个单词（令牌），您正试图为其获取单词向量。手套预训练向量可能没有导致错误的“ng”向量。所以，你需要检查这个词在 Glove 嵌入中是否有一个向量。