Architect Services Installation

Following steps show how to deploy various components of the Architect service and connections to external services. It covers the basic development deployment.

Service architect-api Installation

The core service responsible for handling HTTP API requests and providing simple UI based on Material design. Release version of architect-api is currently available on Pypi, to install it, simply execute:

pip install architect-api

To bootstrap latest development version into local virtualenv, run following commands:

virtualenv -p python3 venv
source venv/bin/activate

git clone git@github.com:cznewt/architect-api.git

cd architect-api

python setup.py install

Or you you can install service by pip architect-api package.

virtualenv venv
source venv/bin/activate

pip install architect-api -e

Initial Setup for UI

Architect-api uses the npm to install its JavaScript dependencies, which are collected by django-npm static files collector. You can install all static nodesj libraries by following commands. More about installing Node.js and NPM can be found at https://www.npmjs.com/get-npm.

apt install npm

cd architect-api

npm install

Architect uses the Bootstrap 4 library wich uses SASS 3.5 style preprocessing. No python SASS interpreter does it well so we need to get the ruby’s gems this time. The static file compress utility uses this ruby binary to perform the processing of SASS styles. You can install the SASS compiler by following commands. More about installing SASS can be found at http://sass-lang.com/install.

apt install gem ruby-dev

sudo gem install sass --no-user-install

sass --version

The saas --version command should return Sass 3.5 or higher.

Now you can collect all your static assets, run following command in architect base dir and sourced.

$ python manage.py collectstatic --noinput

X static files copied to '/python-apps/architect/static', Y unmodified.

Now you can compress and compile your static assets, in architect base dir and sourced run following command.

$ python manage.py compress

Found 'compress' tags in:
    /python-apps/architect/architect/templates/_head.html
    /python-apps/architect/architect/templates/_body.html

Initial Setup for Database

You must synchronise your database content with the current migration scheme, command will create entire schema and apply all the migrations if run for the first time. In architect base dir and sourced run following command.

python manage.py migrate

You need also setup your user credentials if creating a new deployment.

python manage.py createsuperuser

You can install sample metadata fixtures by following command.

$ python manage.py loaddata sample_saltstack

Installed 614 object(s) from 2 fixture(s)

You must set database configuration by settings in architect-api configuration file. Example PostgreSQL settings in architect-api configuration file.

databases:
  default:
    ENGINE: django.db.backends.postgresql_psycopg2
    NAME: architect
    USER: architect
    PASSWORD: password
    HOST: 127.0.0.1
    PORT: 5432

The similar applies for the cache backend, which can be changed to the Memcached backend, for example:

caches:
  default:
    BACKEND: django.core.cache.backends.memcached.MemcachedCache
    LOCATION: 127.0.0.1:11211

Main Configuration File

You provide one YAML configuration file for all settings. The default location is /etc/architect/api.yml.

You can setup basic configuration of database and cache also you can provide defaults for your initial inventories, managers and monitors.

You can override the default location of the configuration file by setting the ARCHITECT_CONFIG_FILE environmental variable to your custom location.

The configuration file currently supports following options:

databases:
  default:
    ENGINE: django.db.backends.postgresql_psycopg2
    ...
caches:
  default:
    BACKEND: django.core.cache.backends.memcached.MemcachedCache
    ...
monitor:
  monitor01:
    name: Dashboard 01
    ...
manager:
  manager01:
    engine: salt
    ...
inventory:
  inventory01:
    engine: reclass
    ...

The databases and caches keys are used in the application settings. But the monitor, manager and inventory configuration settings need to be sychronised to database by management commands in architect base dir and sourced.

$ python manage.py sync_inventories

Inventory "inventory01" resource updated
...

$ python manage.py sync_managers

Manager "manager01" resource updated
...

$ python manage.py sync_monitors

Monitor "monitor01" resource updated
...

You can run the configuration multiple times and update existing resources. The actual resources used are stored in the database and can be changed at the architect’s admin app available at http://127.0.0.1:8181/admin/ after you start the development server.

Look at the the documentation pages for individual inventory, manager or monitor configuration options and installation problems.

Running Development Server

To start development server, in architect base dir and sourced run following command.

$ python manage.py runserver 0.0.0.0:8181

Performing system checks...

System check identified no issues (0 silenced).
January 27, 2018 - 13:12:47
Django version 2.0.1, using settings 'architect.settings'
Starting development server at http://0.0.0.0:8181/
Quit the server with CONTROL-C.

Service architect-worker Installation

The architect relies on standalone workers to perform the tasks asynchronously. For the development environment, you can just simply install redis server to serve as message bus by following command.

apt install redis server

Now you can start running your architect worker instances. The redis is hardcoded and celery can be replaced by airflow, this is up to discussion.

Running development worker

To start development worker, in architect base dir and sourced run following command.

$ celery -A architect worker -l info

 -------------- celery@wst01 v4.1.0 (latentcall)
---- **** -----
--- * ***  * -- Linux-4.10.0-42
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         architect:0x7ff566a38e80
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

[tasks]
  . architect.celery.debug_task
  . get_manager_status_task

[2018-01-27 13:15:55,852: INFO/MainProcess] Connected to redis://localhost:6379//
[2018-01-27 13:15:55,860: INFO/MainProcess] mingle: searching for neighbors
[2018-01-27 13:15:56,880: INFO/MainProcess] mingle: all alone
[2018-01-27 13:15:56,892: INFO/MainProcess] celery@<your-node-hostname> ready.

You should see celery@<your-node-hostname> ready in the output of the command run. If not, check if redis service systemctl status redis-server is running. You need at least one instance of worker running.

Service architect-client Installation

Following steps show how to deploy and configure Architect Client. You need to install client on configuration management servers to integrate the inventory service.

pip install architect-client

Create configuration file /etc/architect/client.yml for client.

project: project-name
host: architect-api
port: 8181
username: salt
password: password

Complete Installation Scripts

Architect API Dependencies

You can install requried services.

#!/bin/bash -ex

export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8

printf "Update Ubuntu ..."
sudo apt-get update -y

printf "Install servers ..."
apt-get -y install memcached redis-server postgresql-9.5

printf "Setup Postgresql ..."
sudo -u postgres psql -c "CREATE DATABASE architect"
sudo -u postgres psql -c "CREATE USER architect WITH PASSWORD 'password'"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE architect TO architect"

Or use local docker-compose.yml to start the same services.

Architect API Core

Install full development environment.

#!/bin/bash -ex

export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8

printf "Update Ubuntu ..."
sudo apt-get update -y

printf "Installing Python 3 dependencies..."
apt-get -y install python-virtualenv python3-dev python3-pip libxml2-dev libxslt1-dev libffi-dev graphviz libpq-dev libssl-dev

printf "Upgrading pip"
pip3 install --upgrade pip

mkdir -p /etc/architect
cat << EOF > /etc/architect/api.yml
databases:
  default:
    ENGINE: django.db.backends.postgresql_psycopg2
    NAME: architect
    USER: architect
    PASSWORD: password
    HOST: 127.0.0.1
    PORT: 5432
caches:
  default:
    BACKEND: django.core.cache.backends.memcached.MemcachedCache
    LOCATION: 127.0.0.1:11211
inventory:
  sample-cluster:
    engine: hier-cluster
    service_class_dir:
    system_class_dir:
    cluster_class_dir:
    class_dir: /srv/architect/mcp/classes
    formula_dir: /srv/salt-formulas/formulas
  sample-deploy:
    engine: hier-deploy
    class_dir: /srv/architect/mcp/classes
    node_dir: /srv/architect/mcp/nodes/sample-deploy
EOF

git clone https://github.com/cznewt/architect-api.git /opt/architect
cd /opt/architect

printf "Installing architect-api code"
virtualenv -p python3 venv
. venv/bin/activate
pip install -r requirements/base.txt
pip install psycopg2-binary
pip install git+https://github.com/salt-formulas/reclass.git@python3

printf "Installing static files tools"
apt-get -y install npm rubygems ruby-dev

sudo gem install sass --no-user-install

npm install

python manage.py collectstatic --noinput
python manage.py compress
python manage.py migrate

# Now you can run following 2 services

# python manage.py runserver 0.0.0.0:8181

# celery -A architect worker -l info

Repository Image Builders

#!/bin/bash -ex

export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8

printf "Update Ubuntu ..."
sudo apt-get update -y

printf "Install Raspberry Pi build dependencies ..."
apt-get install -y debootstrap debian-archive-keyring qemu-user-static binfmt-support dosfstools bmap-tools whois bc crossbuild-essential-armhf

printf "Install BeagleBone build dependencies ..."
apt-get install -y m4 bmap-tools dosfstools rsync git-core kpartx wget parted pv

Sample Hierarchical Inventory

#!/bin/bash -ex

mkdir -p /srv/architect/mcp/classes/deployment
mkdir -p /srv/architect/mcp/classes/service
mkdir -p /srv/architect/mcp/classes/cluster/sample/infra
mkdir -p /srv/architect/mcp/nodes/sample-deploy

if [ ! -d "/srv/architect/mcp/classes/system" ]; then
  git clone https://github.com/Mirantis/reclass-system-salt-model /srv/architect/mcp/classes/system
fi;

if [ ! -d "/srv/salt-formulas" ]; then
  git clone https://github.com/salt-formulas/salt-formulas.git --recursive /srv/salt-formulas
fi;

for i in /srv/salt-formulas/formulas/* ; do
  if [ -d "$i" ]; then
    service=$(basename "$i")
    formula="${service/-/_}"
    if [ -d "/srv/salt-formulas/formulas/$service/metadata/service" ]; then
      if [ ! -L "/srv/architect/mcp/classes/service/$formula" ]; then
        ln -s "/srv/salt-formulas/formulas/$service/metadata/service" "/srv/architect/mcp/classes/service/$formula"
      fi;
    fi;
  fi;
done

cat << EOF > /srv/architect/mcp/classes/cluster/sample/infra/config.yml
classes:
- service.git.client
- system.linux.system.single
- system.linux.system.repo.mcp.salt
- system.salt.master.pkg
parameters:
  _param:
    salt_master_host: 127.0.0.1
    salt_master_environment_name: dev
    salt_master_environment_repository: https://github.com/salt-formulas
    salt_master_environment_revision: master
    salt_minion_ca_authority: salt_master_ca
  salt:
    master:
      user:
        salt:
          permissions:
          - '.*'
          - '@local'
          - '@wheel'   # to allow access to all wheel modules
          - '@runner'  # to allow access to all runner modules
          - '@jobs'    # to allow access to the jobs runner and/or wheel modules
      engine:
        architect:
          engine: architect
          project: sample-deploy
          host: 127.0.0.1
          port: 8181
          username: architect
          password: password
EOF

cat << EOF > /srv/architect/mcp/classes/deployment/sample-deploy.yml
parameters:
  _param:
    cluster_name: sample-deploy
    cluster_domain: sample.deploy
EOF

cat << EOF > /srv/architect/mcp/nodes/sample-deploy/cfg01.sample.deploy.yml
classes:
- cluster.sample.infra.config
- deployment.sample-deploy
parameters:
  _param:
    linux_system_codename: xenial
  linux:
    system:
      name: cfg01
      domain: sample.deploy
EOF