[Bug] api client multiprocessing usage incompatible with AWS lambda

TL;DR

Fixes: upgrade to langchain>=0.1.17 / langchain-pinecone>=0.1.0 / langchain-community>=0.0.37, which no longer hard-code asyncreq, then pass async\req=False; or wrap a concurrent.futures.ThreadPoolExecutor and assign it to index.\_pool (reporter's workaround).

Steps

1. [kazzmir (reporter), closed by daverigby (maintainer)]: Root cause: langchain hard-coded async_req=True, which makes the pinecone client spawn a multiprocessing pool whose _multiprocessing.SemLock is not implemented on AWS Lambda (python3.11). Fixes: upgrade to langchain>=0.1.17 / langchain-pinecone>=0.1.0 / langchain-community>=0.0.37, which no longer hard-code asyncreq, then pass `asyncreq=False; or wrap a concurrent.futures.ThreadPoolExecutor and assign it to index._pool` (reporter's workaround).

When to use

You are seeing this: Issue pinecone-io/python-sdk#257 (closed, 5 comments): ### Is this a new bug in the Pinecone Python client? - [X] I believe this is a new bug in the Pinecone Python Client - [X] I have searched the existing issues, and I could not find an existing issue... Use this skill when you run into "[Bug] api client multiprocessing usage incompatible with AWS lambda".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: langchain 0.1.17, langchain-pinecone 0.1.0, langchain-community 0.0.37. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda

Asked in gh:pinecone-io/python-sdk#257 (https://github.com/pinecone-io/python-sdk/issues/257): [Bug] api client multiprocessing usage incompatible with AWS lambda. The issue report: Issue pinecone-io/python-sdk#257 (closed, 5 comments): ### Is this a new bug in the Pinecone Python client?

• [X] I believe this is a new bug in the Pinecone Python Client
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Using the python api client ultimately results in an invocation of _multiprocessing.SemLock, which is not implemented on AWS lambda using a python3.11 runtime. Related stackoverflow: https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda

Expected Behavior

Using the python API client should work The verified answer: [kazzmir (reporter), closed by daverigby (maintainer)]: Root cause: langchain hard-coded async_req=True, which makes the pinecone client spawn a multiprocessing pool whose _multiprocessing.SemLock is not implemented on AWS Lambda (python3.11). Fixes: upgrade to langchain>=0.1.17 / langchain-pinecone>=0.1.0 / langchain-community>=0.0.37, which no longer hard-code asyncreq, then pass `asyncreq=False; or wrap a concurrent.futures.ThreadPoolExecutor and assign it to index._pool` (reporter's workaround).