Python AWS Lambda Monorepo — Part 2: Custom Package Sharing

  • bash 5.0 (terminal shell)
  • pip 19.3.1
  • make 3.81
The Lost World (1997) | Universal Studios/Amblin

Let’s get this moveable feast underway.

Custom Python Package

Modules

Let’s pause for a moment to review some Python concepts: modules and packages. A module is a Python file containing functions, classes and/or statements definitions that relate logically. In reality, any Python file can be used as a module. Modules are added to our code by using the import statement followed by the name of the module .py file. The Python interpreter looks for modules in the following locations:

  1. The directory where the file is being run (and its subdirectories)
  2. The directories specified by thePYTHONPATH env variable
  3. The dependency installations directory
# example.py
def greet(name):
print(f'Hello {name}')
# import1.py
import example
example.greet('Snake')
>> Hello Snake
# import2.py
from example import greet
greet('Otacon')
>> Hello Otacon

Packages

A package is a grouping of modules in a directory with a specified hierarchy. A package can contain a single module or multiple sub-modules and sub-packages. Say we have the following example package and python scripts:

.
├── /my_package
| └── example.py
├── import1.py
├── import2.py
├── import3.py
├── import4.py
...
# import1.py
from my_package import example
example.greet('Leonardo')
>> Hello Leonardo
# import2.py
from my_package.example
import greetgreet('Donatello')
>> Hello Donatello
# import3.py
import my_package.example as pkg
pkg.greet('Michelangelo')
>> Hello Michelangelo
# import4.py
from my_package import example as pkg
pkg.greet('Raphael')
>> Hello Raphael
.
├── /my_package
| ├── __init__.py
| └── example.py
├── import1.py
...
# __init__.py
import my_package.example
print(f'Package name: {__name__}')
for x in range(2):
print(x)
# import1.py
import my_package
my_package.example.greet('Lord Vader')
>> Package name: my_package
>> 0
>> 1
>> Hello Lord Vader
.
├── /my_package
| ├── __init__.py
| ├── setup.py
| └── example.py
...
# setup.py
import setuptools
setuptools.setup(
name = "my_package",
version = "0.0.1",
author = "@bombillazo",
author_email = "example@bombillazo.com",
description = ("Brief description of my_package"),
license = "MIT",
scripts=['example.py'],
python_requires=">=3.7",
)
.
├── /my_package
| ├── build
| | └── scripts-3.7
| | └── example.py
| ├── __init__.py
| ├── setup.py
| └── example.py
...

Custom Package

Now that we know the basics of how modules and packages work, let’s create our own. Start by creating the packages directory inside our root project folder:

.
└── /packages
.
├── /packages
| └── /dinosaur
| ├── /dinosaur
| | ├── __init__.py
| | └── core.py
| └── setup.py
...
...
packages=setuptools.find_packages(),
...
# __init__.py
from dinosaur.core import Dinosaur
.
├── /packages
├── /services
| ├── /create_dinosaur
| ├── /create_hybrid_dinosaur
| └── /fight_dinosaur
...
.
├── /packages
├── /services
| ├── /create_dinosaur
| | ├── main.py
| | └── requirements.txt
| ...
...
# requirements.txt
./packages/dinosaur
from dinosaur import Dinosaur...new_dinosaur = Dinosaur(
name=dino_name,
diet=dino_diet,
period=dino_period,
weight=dino_weight,
armor=dino_armor,
hybrid=dino_hybrid
)
dynamodb = boto3.resource('dynamodb')
dinosaurs_table = dynamodb.Table('dinosaurs')
...response = dinosaurs_table.put_item(
Item={
'name': new_dinosaur.name,
'diet': new_dinosaur.diet,
'period': new_dinosaur.period,
'attack': new_dinosaur.attack,
'defense': new_dinosaur.defense,
'life': new_dinosaur.life,
'info': {
'weight': new_dinosaur.weight,
'armor': new_dinosaur.armor,
'nature': new_dinosaur.nature,
'hybrid': new_dinosaur.hybrid
}
}
)
dinosaur_list = dinosaurs_table.scan(
FilterExpression=Attr('info.hybrid').eq(False),
ProjectionExpression='#name',
ExpressionAttributeNames={ '#name': 'name' }
)['Items']
...dino_1 = dinosaurs_table.query(
KeyConditionExpression=Key('name').eq(dinosaur_pair[0]['name'])
)['Items'][0]

Custom Package Install

We set up our Lambda functions save for one important part: our Lambda functions are missing the dinosaur package. Remember how Python looks for modules? Our dinosaur package is not in any of those locations. We need a way to distribute our custom package to each lambda.

.
├── /packages
├── /services
└── Makefile
make install-packages LAMBDA=function_name
#Makefile
install-packages:
FILE=services/${LAMBDA}/requirements.txt; \
if [[ -f "$$FILE" ]]; then \
pip install -r $$FILE --target services/${LAMBDA}/packages/ --find-links ./packages --upgrade; \
fi
pip install -r $$FILE --target services/${LAMBDA}/packages/ --find-links ./packages --upgrade;
make install-packages LAMBDA=create_dinosaur
.
├── /packages
├── /services
| ├── /create_dinosaur
| | ├── /packages
| | | ├── /dinosaur
| | | | ├── /__pycache__
| | | | ├── __init__.py
| | | | └── core.py
| | | └── /dinosaur-1.0.0.dist-info
| | ├── main.py
| | └── requirements.txt
| ...
...

Co-founder and CTO of Hecdemi / Hyperion. Computer Engineer and entrepreneur from Puerto Rico 🇵🇷 Interested in combining tech, business, and product design.

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store