Introduction

Pyaella is a development toolkit designed to make it easier to implement enterprise-class systems. Its primary goal is to automate and speed prototyping and implementation iterations, as much as possible. Pyaella includes tools for metacoding and generation, enabling a fast turn-over of ideas and models, so a developer can focus more on design and business logic through many rapid iterations.

With Pyaella you can design a full backend for a site or distributed application, defining logical models, business rules, data models with complete relational integrity, SQL and RBAC control... all within a single ideal day.

Philosophy

The philosophy behind Pyaella is simple: Make it painless, if not enjoyable to go back into the kitchen, so to speak, and start over; to continually improve upon design, usability, stability, team velocity, and efficiency.

Beginnings

This set of tools and patterns was initially put together to build custom geo-spatial backends and mobile applications, to leverage the power of PostgreSQL and PostGIS in an agile development environment. To do that Pyaella needed to describe business domains and databases, but it also needed to include a flexible business layer, with great dependency injection capabilities and a lightweight DSL. Pyaella’s logical domain layer uses easy to configure code dependencies, dynamically created metaclasses and classes, and SQLAlchemy / GeoAlchemy2 to represent and persist relational data. But to make business systems viable, Pyaella or apps developed with it need to run on modern infrastructures like Amazon and Heroku, so build and deployment strategies and utilities needed to be included.

The same development patterns presented themselves and were prime candidates for automation and metacode.

By defining a single application domain file describing logical models, relationships, rules, aspects, and injections... a new database, ORM, with logical models and business rules, an API, and a complete uWSGI supporting Pyramid application can be up in a matter of minutes.

What Pyaella is not

  • Pyaella is not a web framework. It relies on Pyramid
  • Pyaella is not a templating or presentation system. It relies on Mako and Foundation
  • Pyaella is not a NoSQL solution. It relies on PostgreSQL
  • Pyaella is not an ORM or SQL expression toolkit. It relies on SQLAlchemy
  • Pyaella is not a messaging system. It relies on RabbitMQ and Celery
  • Pyaella is not a remoting / RMI layer. It relies on Pyro

Pyaella, not just for web apps

Python is a batteries included language. The apps being developed using Pyaella at Miga Col.labs require so many different packages, libraries, formidable and fundimental technologies we consider the entire stack packed with fuel cells.

  • Integrated white-labeling based on Foundation
  • Support and admin utils for the cloud at Heroku
  • Amazon S3 integration AWS
  • Geo-spatial logical models, functions, and resources using GEOS
  • Complete PostGIS integration
  • Easy-to-develop i18n and l10n
  • Numpy, SciPy
  • Heavyweight and lightweight Messaging
  • Automated workflow management
  • Mobile agents and negociated automation

Quicklook

The core components to Pyaella include a “domain” file that describes logical models and their rules, relationships, and the system in general. This includes information on which metaclasses and superclass to use, how classes are organized and compiled, and any required dependency data to be injected into the system at runtime.

Using compilation, install, and refresh tools, this domain file is parsed to create types, classes, models, object-relational mappings, databases, and configures the application runtime, et cætera.

After installing Pyaella and setting up the environment, we can quickly create an example application and RDMBS by creating a new domain.pyl file.

Defining logical model rules

Here we create a User, with different User Types.

User:
    Fields:
        id: Column(BigInteger,
            Sequence('users_id_seq', start=1011011011),
            CheckConstraint('id<11111111111'),
            server_default=text("nextval('users_id_seq')"),
            primary_key=True)
        user_name: Column(String(50), nullable=False, unique=True)
        email_address: Column(String(255), nullable=False, unique=True)
        password: Column(String(128),
            CheckConstraint('"char_length"(password)>126'), nullable=False)
        is_active: Column(Boolean, default=True)
        first_name: Column(String(64), nullable=False)
        last_name: Column(String(64), nullable=False)
        country_code: Column(String(3), nullable=False)
        open_id: Column(String(128))
        auth_code: Column(String(8))
    Relations:
        user_types: relationship('UserXUserTypeLookup')

UserTypeLookup:
    Fields:
        name: Column(String(24), unique=True, nullable=False)
        description: Column(String(64), nullable=False)
    Values:
        name: [
            Sys,
            Dev,
            Admin,
            General
        ]
        description: [
            System Administrator,
            Developer,
            Administrator,
            General User
        ]
    Rules:
        Unique: [name]

UserXUserTypeLookup:
    Relations:
        user_id: Column(BigInteger, ForeignKey('users.id'), primary_key=True)
        user_type_id: Column(Integer, ForeignKey('user_type_lu.id'), primary_key=True)

The above defines a User logical model with numerous Fields. In this example we create a SQLAlchemy Column type directly. Any valid Column construction is supported, allowing us to define a logical model with parameters, but maintaining a decoupled relationship to an ORM and its underlying persistence layer.

UserTypeLookup is a defined as a “lookup” model, where each unique name has a description. Each name:description pair defined in a Pyaella “Lookup” model is inserted/upserted into the RDBMS when compiled or upon start-up.

We can associate a User with a UserType by explicity using an “association” or “cross” table. Depending on the case and business rules, this explicit defination may not be required and can be created automatically.

These models are saved to a “domain” file. This domain file is used when designing and implementing the system, generating classes, iterating over implementation details and ideas, and finally when configuring the runtime.

Warning

Though these logical models can technically be created without any need for persistence and be decoupled from any ORM, that wouldn’t make much sense to 99.3% (+/-) of the applications built.

Compiling the logical domain

Pyaella compiles the domain into supporting modules using pyaellac.

>>> python -m pyaella.meta.pyaellac -o models.py -d domain.pyl

The pyaellac compilation will output a models.py module that will include decoupled classes that will use the associated domain file during runtime.

__autogen_date__ = "2013-08-19 05:43:07.536073"
__schema_file__ = os.path.join(os.path.dirname(__file__), "domain.pyl")
MODEL_SCHEMA_CONFIG = __borg_lex__('ModelConfig')(parsable=__schema_file__)

__all__ = [
    "UserTypeLookup",
    "UserXUserTypeLookup",
    "User"
]

class UserTypeLookup(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)


class UserXUserTypeLookup(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)


class User(PyaellaDataModel):
    __metaclass__ = PyaellaDataModelMetaclass

    def __init__(self, base=Base, **kw):
        PyaellaDataModel.__init__(self, base=base, **kw)

You can create the classes and the logical models by hand, but by using pyaellac its easier to manage automatic migrations when logical models and data model schemas change and need to be represented in a database.

Once the domain has been compiled, and we have a Pyaella scaffold built, we can edit an appcfg file to defile the runtime.

App:
    AppId: 1600
    DomainId: 1357913579135791
    UploadDirectory: uploads/
    AssetDepot: s3://asset_depot
    AsyncFamily: Threading
    BackgroundModules: [bgprocs]
Web:
    ScanPackage: miga
    TemplateDir: miga/templates
    StaticDirs:
        assets: miga/assets
        player: miga/player
        css: miga/css
        scripts: miga/scripts
    SiteName: localhost
Resources:
    Database:
        User: postgres
        Password:
        Host: localhost
        Port: 5432
        Schema: pyaella-demo
        CreateTables: True
    ORM:
        PoolSize: 20
        MaxOverflow: 10
        ConvertUnicode: True
        Echo: True
    Contexts: miga.contexts
    Models: miga.models
    Schema: miga/domain.pyl

This appcfg example above sets up the runtime, and is displayed here for the quicklook example. The details of application configuration is document elsewhere.

Creating the database for the first time

Running the dbinstall will create a brand new database called pyaella-demo that supports Pyaella, its custom types and functions.

python -m pyaella.server.dbinstall -U postgres -O postgres --host localhost --port 5432 --contrib-dir /opt/local/share/postgresql93/contrib/postgis-2.0 --db pyaella-demo --appcfg appcfg.yaml

The pyaella.server.dbinstall takes arguments to specify the user, the host machine, port, and the new database’s name. It also takes an argument as the path to the PostgreSQL contribs directory if it exists in a custom location.

The dbinstall script uses the appcfg.yaml file, so that must be supplied too.

Once the database has been created and is running, we can connect a Pyaella application easily. An application could be a client library, daemon, or a site, &c. Any Pyaella application requires the domain file and the appcfg file.

Creating a Pyaella application for Pyramid

Installation

Definition

Configuration

Deployment

A Pyramid view_callable example

This is an example snippet directly from a Miga project

Query for a User of UserType using Pyaella’s SQLAlchemySessionFactory and the SQLAlchemy relationship on User for UserXUserTypeLookup

with SQLAlchemySessionFactory() as session:
    res_prxy = session.query(~User).filter((~User).user_types.name=='admin').all()

The following does an explicit join and lookup to find an admin

@view_config(
    name='foundme',
    request_method='GET',
    context='miga:contexts.Geo',
    renderer="foundme.mako")
def foundme(request):
    """ render map of user and nearest droplets
        foundme/lng=10.3031649469030580/lat=43.5311867249821205
    """
    auth_usrid = authenticated_userid(request)

    with SQLAlchemySessionFactory() as session:

        U, UT, UxUTL = ~User, ~UserTypeLookup, ~UserXUserTypeLookup
        user = (session.query(U, UT, UxUTL)
                    .filter(U.email_address==auth_usrid)
                    .filter(UxUTL.user_id==U.id)
                    .filter(UxUTL.user_type_id==UT.id)
                    .filter(UT.name=='admin')).first()

Using the default API

Each logical model defined in the domain can be accessible through Pyaella’s RESTlike API, an application programming interface that presents method signature-like URIs to GET, POST, PUT, DELETE entities.

Warning

Pyaella uses JSON and JSONP exclusively, and does not support “web services” or any XML bloat-ware approaches to semantic systems or Web3.0

A simple GET:

http://camp.awesome.sauce.com/m/edit/User/id=5

In the above example the m/ traversal resource is called with the resource name edit for the subpath User/id=5. User in the logical model, which will be used to retrieve a User entity, using the ORM to access the database, filtering for the User with the id “5”. GETS are used for retrieval, and not searches.

A search for entities:

http://camp.awesome.sauce.com/m/search/User/first_name=Xef

The search resource will return, if any, all Users with the first name “Xef”.

So what did we just do?

Without too much code and writing two configuration files, one a “domain” file and another a “appcfg” file Pyaella built a Pyramid application, a PostgreSQL database, and presents an RESTlike API to the logical and data models. Of course many details were glossed, but most other code and organization required is boilerplate and documented elsewhere.

Roadmap ahead

Pyaella’s goal is to mature into a complete stack of development tools to help rapidly cook-up new distributed applications, large infrastructure back-ends, mobile applications, and automation systems; to help the creative process designing, implementing, and testing immediately. With good tools a team can prove a new idea without having to get bogged down with repetitive work, tasks that should be automated, and the headaches of starting from scratch.

Contents:

Indices and tables