AWS Lambda Runtime Compatibility Issue on Native Library

No module named ‘_cffi_backend’”

Issue In Local Development Environment

When building serverless applications, often time we use SAM to deploy and develop lambda functions in the local environment. Most of our engineers in the firm using macOS or Windows, which is a different runtime environment as Lambda runtime environment on AWS. Just take one of my examples, I was trying to encrypt/decrypt documents through python cryptography  The program can run perfectly on my local windows environment, then I deployed everything from SAM. When I ran the lambda function on the AWS console, I was getting the following error.

{ 
  "errorMessage": "Unable to import module 'python_handler': No module named '_cffi_backend'", 
  "errorType": "Runtime.ImportModuleError" 
}

Native Library Package Incompatibility

At first, I thought it was an import issue, so I try to package it again. The issue was still present. 

After doing research on the internet, I found a StackOverflow discussion and a technical blog. I found it could be a packaging issue since AWS Lambda is running on a Linux environment, but I packaged it through windows or MacOs.  The issue could also surge when testing the code on lambda container through sam invoke or other equivalent methods.

Root Cause of Lambda Incompatibility

After I dug deep into the problem, it was the package generated on the Windows platform with the pip command. The libraries, native to OS, were compiled for the windows or Mac operation system. 

In my case, cryptography is a platform-specific library. So, if I run my function on AWS environment, I got “Unable to import module ‘python_handler’: No module named ‘_cffi_backend’”

This problem is not only for the cryptography library but also for all python libraries written in the C language. Such as Pillow(image processing), pandas(scientific lib), sklearn(Machine learning) and etc.  Because these compiled binary c libs are platform-dependent. 

We could deduce that error not only occurs in Python but also in Nodejs. Because Nodejs is also a platform-dependent language. 

After knowing the root cause, the solution is relatively easy.

The solutions for Lambda Incompatibility

There are at least two solutions for us.

Solution 1: Standardized building environment.

As we know the issue comes from the discrepancy between the building environment and the lambda runtime environment. So, the easiest way is to change the building environment to the same one as the running environment.

We can launch an Amazon Linux instance on the AWS EC2 platform. And then build and package our lambda code there. 

In this way, the environmental discrepancy is eliminated. I believe it’s a good practice for the CI/CD process but lots of overhead for a local development environment.

Solution 2: Containerization(Docker)

As we mentioned the previous solution has lots of overhead for local development. One can simulate the environment with containerization technology. Docker is able to provide us perfectly isolated Linux environment. We could use an Amazon Linux compatible image to build and package.

Sample script

mkdir -p /tmp/python/lib/python3.7/
cp ./requirements.txt /tmp/
cd /tmp/
# this image is compatible with the lambda runtime.
docker run \
-v "$PWD":/var/task \
"lambci/lambda:build-python3.7" \
/bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"
zip -r package.zip python > /dev/null

Why This Class of Error Keeps Recurring

The _cffi_backend failure is one instance of a general rule worth internalising: a Python wheel containing compiled C extensions is built against a specific platform, CPython version, and C library. Install it on macOS or Windows and pip fetches a wheel for that platform. Ship that directory to Lambda, which runs Amazon Linux, and the interpreter loads a shared object built for the wrong system.

The symptom is misleading, which is what costs the time. ModuleNotFoundError: No module named '_cffi_backend' reads like a missing dependency, so the instinct is to reinstall the package — and the package is present. What is missing is a loadable build of its native component. The same failure shows up behind cryptography, psycopg2, numpy, pandas, Pillow, and lxml, all of which ship compiled extensions.

A quick way to confirm the diagnosis before changing anything: look inside the deployment package for files ending in .so. If the filenames carry a platform tag that is not Linux — darwin, win_amd64 — the artefact was built on the wrong machine and no amount of reinstalling will fix it.

Two More Options Worth Knowing

Beyond the standardized build host and the Docker approach above, two later additions to Lambda change the calculus.

Lambda layers let you build the native dependencies once, publish them as a versioned layer, and attach that layer to any function that needs it. The build still has to happen in a Linux environment, so this does not replace the techniques above — but it does mean the build happens once per dependency set rather than once per function, and functions sharing a dependency share a single artefact. This also keeps the function’s own deployment package small enough to remain editable in the console.

Container image packaging takes the idea further. Rather than zipping dependencies, package the function as a container image built FROM an AWS base image for your runtime. The build environment and the execution environment are then the same artefact by construction, which removes the entire class of mismatch rather than working around it. It also raises the size ceiling substantially, which matters for dependency sets that will not fit in a zip. The trade-off is a container build and an ECR repository in the pipeline.

If you are choosing today: reach for a layer when several functions share the same native dependencies, and for a container image when the dependency set is large or the mismatch keeps recurring despite a standardized build.