Introduction

Pyaella is a development toolkit designed to make it easier to implement enterprise-class systems. The goal is to help rapidly prototype complex systems, quickly tear-down, build-up iterations of an idea. Pyaella includes tools for metaclasses, metacoding, dependency injection, to simplify desiging business models, logical models, persistence and access.

With this toolkit its possible to rapidly implement a complete backend; defining logical, data and business models with relational integrity and role-based access control.

Warning

This is preliminary documentation and will be constantly changing and improving. Good documentation is hard, and though this project has been under development and active and in production for a few years, the docs are still far behind.

Design Philosophy

The philosophy is simple: Make it painless to start over, to continually improve upon design, usability, stability, and scalability.

Beginnings

This set of tools was initially put together to build custom backends for geospatial analysis and mobile applications.... to leverage the power of PostgreSQL and PostGIS in an agile development environment. To do that we needed to describe and define business domains quickly, using dependency injection and a lightweight DSL. Pyaella’s logical domain layer uses dynamically created metaclasses and classes, and SQLAlchemy and GeoAlchemy2 as an ORM to the relational data stored in PostgreSQL.

By defining a single domain file describing models, relationships, rules, aspects, and mixins, a new database, an ORM with logical models, business objects and rules, an API, a complete uWSGI supporting Pyramid application can be up in a matter of... well, very quickly.

In Practice

MIGA has been using this code base to prototype and implement numerous backends for projects ranging from gaming analytics to real estate asset management systems.

What Pyaella is not

  • Pyaella is not a web framework. It relies on Pyramid
  • Pyaella is not a templating system. It relies on Mako
  • Pyaella is not a HTML/CSS framework. It relies on either Foundation or MaterializeCSS
  • Pyaella is not a NoSQL solution. It relies on PostgreSQL and Redis
  • Pyaella is not an ORM or SQL expression toolkit. It relies on SQLAlchemy
  • Pyaella is not a messaging system. It relies on RabbitMQ and Celery
  • Pyaella is not a remoting / RMI layer. It relies on Pyro

Pyaella, not just for web apps

  • Pyaella and Kivy
  • Pyaella and Qt
  • Support and admin utils for the cloud at Heroku
  • Amazon S3 integration AWS
  • Geo-spatial logical models, functions, and resources using GEOS
  • Complete PostGIS integration
  • Easy-to-develop i18n and l10n
  • Numpy, SciPy
  • Both Heavyweight and lightweight Messaging
  • Automated workflow management
  • Mobile agents and negociated automation

Quicklook

The core components to Pyaella include a “domain” file that describes the system. This includes information on which metaclasses and superclass to use, how classes are organized and compiled, and any dependency data to be injected at runtime.

Using compilation, install, and refresh tools, the domain file is parsed to create types, classes, models, object-relational mappings, databases, and configures the application runtime, et cætera.

Defining logical models

Here we create a User, with different User Types.

User:
    Fields:
        id: Column(BigInteger,
            Sequence('users_id_seq', start=1011011011),
            CheckConstraint('id<11111111111'),
            server_default=text("nextval('users_id_seq')"),
            primary_key=True)
        user_name: Column(String(50), nullable=False, unique=True)
        email_address: Column(String(255), nullable=False, unique=True)
        password: Column(String(128),
            CheckConstraint('"char_length"(password)>126'), nullable=False)
        is_active: Column(Boolean, default=True)
        first_name: Column(String(64), nullable=False)
        last_name: Column(String(64), nullable=False)
        country_code: Column(String(3), nullable=False)
        open_id: Column(String(128))
        auth_code: Column(String(8))
    Relations:
        user_types: relationship('UserXUserTypeLookup')

UserTypeLookup:
    Fields:
        name: Column(String(24), unique=True, nullable=False)
        description: Column(String(64), nullable=False)
    Values:
        name: [
            Sys,
            Dev,
            Admin,
            General
        ]
        description: [
            System Administrator,
            Developer,
            Administrator,
            General User
        ]
    Rules:
        Unique: [name]

UserXUserTypeLookup:
    Relations:
        user_id: Column(BigInteger, ForeignKey('users.id'), primary_key=True)
        user_type_id: Column(Integer, ForeignKey('user_type_lu.id'), primary_key=True)

The above defines a User logical model with numerous Fields. In this example we create a SQLAlchemy Column type directly. Any valid Column construction is supported, allowing us to define a logical model with parameters, but maintaining a decoupled relationship to an ORM and its underlying persistence layer.

UserTypeLookup is a defined as a “lookup table” style model, where each unique name has a description. Each name/description pair defined in this style model is upserted into the table when compiled or upon start-up.

We can associate a User with a UserType by explicity using an association table. Depending on the case and business rules, this explicit defination may not be required and can be created automatically.

These models are saved to a “domain” file. This domain file is used when designing and implementing the system, generating classes, iterating over implementation details and ideas, and finally when configuring the runtime.

Compiling the logical domain

Pyaella compiles the domain into supporting modules using pyaellac.

>>> python -m pyaella.metacode.pyaellac -d domain.plr -o models.py

The pyaellac compilation will output a models.py module that will include decoupled classes that will use the associated domain file during runtime.

__autogen_date__ = "2013-08-19 05:43:07.536073"
__schema_file__ = os.path.join(os.path.dirname(__file__), "domain.pyl")
MODEL_SCHEMA_CONFIG = __borg_lex__('ModelConfig')(parsable=__schema_file__)

__all__ = [
    "UserTypeLookup",
    "UserXUserTypeLookup",
    "User"
]

class UserTypeLookup(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)


class UserXUserTypeLookup(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)


class User(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)

You can create the classes and the logical models by hand, but by using pyaellac its easier to manage automatic migrations when logical models and data model schemas change and need to be represented in a database.

Once the domain has been compiled, and we have a Pyaella scaffold built, we can edit an appcfg file to defile the runtime.

App:
    AppId: 1600
    DomainId: 1357913579135791
    UploadDirectory: uploads/
    AssetDepot: s3://asset_depot
    AsyncFamily: Threading
    BackgroundModules: [bgprocs]
Web:
    ScanPackage: miga
    TemplateDir: miga/templates
    StaticDirs:
        assets: miga/assets
        player: miga/player
        css: miga/css
        scripts: miga/scripts
    SiteName: localhost
Resources:
    Database:
        User: postgres
        Password:
        Host: localhost
        Port: 5432
        Schema: pyaella-demo
        CreateTables: True
    ORM:
        PoolSize: 20
        MaxOverflow: 10
        ConvertUnicode: True
        Echo: True
    Contexts: miga.contexts
    Models: miga.models
    Schema: miga/domain.pyl

This appcfg example above sets up the runtime, and is displayed here for the quicklook example.

Creating the database for the first time

Running the dbinstall will create a brand new database called pyaella-demo that supports Pyaella, its custom types and functions.

python -m pyaella.server.dbinstall -U postgres -O postgres --host localhost --port 5432 --contrib-dir /opt/local/share/postgresql93/contrib/postgis-2.0 --db pyaella-demo --appcfg appcfg.yaml

The pyaella.server.dbinstall takes arguments to specify the user, the host machine, port, and the new database’s name. It also takes an argument as the path to the PostgreSQL contribs directory if it exists in a custom location.

The dbinstall script uses the appcfg.yaml file.

Creating a Pyaella application for Pyramid

Installation

Definition

Configuration

Deployment

A Pyramid view_callable example

This is an example snippet from a Miga project

Query for a User of UserType using Pyaella’s SQLAlchemySessionFactory and the SQLAlchemy relationship on User for UserXUserTypeLookup

with SQLAlchemySessionFactory() as session:
    res_prxy = session.query(~User).filter((~User).user_types.name=='admin').all()

The following does an explicit join and lookup to find an admin

@view_config(
    name='foundme',
    request_method='GET',
    context='miga:contexts.Geo',
    renderer="foundme.mako")
def foundme(request):
    """ render map of user and nearest droplets
        foundme/lng=10.3031649469030580/lat=43.5311867249821205
    """
    auth_usrid = authenticated_userid(request)

    with SQLAlchemySessionFactory() as session:

        U, UT, UxUTL = ~User, ~UserTypeLookup, ~UserXUserTypeLookup
        user = (session.query(U, UT, UxUTL)
                    .filter(U.email_address==auth_usrid)
                    .filter(UxUTL.user_id==U.id)
                    .filter(UxUTL.user_type_id==UT.id)
                    .filter(UT.name=='admin')).first()

Using the default API

Each logical model defined in the domain can be accessible through Pyaella’s RESTlike API, an application programming interface that presents method signature-like URIs to GET, POST, PUT, DELETE entities.

Warning

Pyaella uses JSON and JSONP exclusively, and does not support “web services” or any XML bloat-ware

A simple GET:

http://camp.awesome.sauce.com/m/edit/User/id=5

In the above example the m/ traversal resource is called with the resource name edit for the subpath User/id=5. User in the logical model, which will be used to retrieve a User entity, using the ORM to access the database, filtering for the User with the id “5”. GETS are used for retrieval, and not searches.

A search for entities:

http://camp.awesome.sauce.com/m/search/User/first_name=Xef

The search resource will return, if any, all Users with the first name “Xef”.

So what did we just do?

Without too much code and writing two configuration files, one a “domain” file and another a “appcfg” file Pyaella built a Pyramid application, a PostgreSQL database, and presents an RESTlike API to the logical and data models. Of course many details were glossed, but most other code and organization required is boilerplate and documented elsewhere.

Roadmap ahead

Pyaella’s goal is to mature into a complete stack of development tools to help rapidly cook-up new distributed applications, large infrastructure back-ends, mobile applications, and automation systems; to help the creative process designing, implementing, and testing immediately. With good tools a team can prove a new idea without having to get bogged down with repetitive work, tasks that should be automated, and the headaches of starting from scratch.

Contents:

Indices and tables