大集合的 Firestore DeadlineExceeded 异常

Question

我正在尝试从 Google Firestore 读取更大的集合以进行测试和存档。当我尝试从包含超过 6k 文档的集合中获取所有文档时，我遇到了一些有趣的错误。

朴素的 Python 解决方案

我的第一次尝试是使用 Python google-cloud-firestore（0.30.0 版）库。

source_client = firestore.Client()

source = source_client.collection(collection)

source_data = source.get()

counter = 0

for f in source_data:

    app.logger.info(f.id)

    counter += 1

    if counter % 100 == 0:

        app.logger.info('%s %d', datetime.now(), counter)

    app.logger.info('%s Finally read all %d documents', datetime.now(), counter)

给出以下输出：

INFO:flask.app:2018-11-08 09:49:03.923795 6400  

INFO:flask.app:2018-11-08 09:49:04.115410 6500  

... 

INFO:flask.app:2018-11-08 09:49:03.923795 6400

INFO:flask.app:2018-11-08 09:49:04.115410 6500

WARNING:flask.app:2018-11-08 09:49:04.128478 copy brocken by exception

Traceback (most recent call last):

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__

    return self.wsgi_app(environ, start_response)

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app

    response = self.handle_exception(e)

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception

    reraise(exc_type, exc_value, tb)

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise

    raise value

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app

    response = self.full_dispatch_request()

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception

    reraise(exc_type, exc_value, tb)

这似乎是由配额引起的。即使我在这里看不到它。这似乎是基于时间的，因为当我在元素之间进行少量睡眠时，我的吞吐量会减少，并在 ~50 秒后出现异常。

Answer 1

在 firebase 支持团队的帮助下，我们发现 python 客户端 api 确实存在错误。下一个版本中有一个错误修正。很可能它将使 python 库能够按 documentid 排序，因此使用start_after().到那时你有两种可能的解决方案：使用另一个字段进行排序和使用start_after()将 node.js 库与分页一起使用，例如：var db = admin.firestore();admin.firestore().settings({ timestampsInSnapshots: true });function readNextPage(lastReadDoc) {  let query = db    .collection(collection)    .orderBy(admin.firestore.FieldPath.documentId())    .limit(100);}

Answer 2

在我的例子中，我在获取整个集合时遇到了这个错误。它甚至不是那么大的集合，但我猜集合中的文件很大。我做了分页更新。这是一个节点 firebase 函数：let lastReadDoc = false;let lastDoc: string = '';&nbsp; const limitRecordCount = 10;&nbsp; do {&nbsp; &nbsp; await db&nbsp; &nbsp; &nbsp; .collection('something/' + somethingId + '/somethingcollection')&nbsp; &nbsp; &nbsp; .orderBy('id')&nbsp; &nbsp; &nbsp; .limit(limitRecordCount)&nbsp; &nbsp; &nbsp; .startAfter(lastDoc)&nbsp; &nbsp; &nbsp; .get()&nbsp; &nbsp; &nbsp; .then((snapshot: any) => {&nbsp; &nbsp; &nbsp; &nbsp; let counter = 0;&nbsp; &nbsp; &nbsp; &nbsp; snapshot.docs.forEach((doc: any) => {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; const docData = doc.data();&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if (lastDoc !== docData.id) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; lastDoc = docData.id;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; counter = counter + 1;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; // ***********************&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; // business logic per doc here&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; // ***********************&nbsp; &nbsp; &nbsp; &nbsp; });&nbsp; &nbsp; &nbsp; &nbsp; if (counter < limitRecordCount) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; lastReadDoc = true;&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; })&nbsp; &nbsp; &nbsp; .catch((err: any) => {&nbsp; &nbsp; &nbsp; &nbsp; lastReadDoc = true;&nbsp; &nbsp; &nbsp; &nbsp; console.log('Error getting booking documents', err);&nbsp; &nbsp; &nbsp; });&nbsp; } while (lastReadDoc === false);