Architect Services Installation¶
Following steps show how to deploy various components of the Architect service and connections to external services. It covers the basic development deployment.
Service architect-api
Installation¶
The core service responsible for handling HTTP API requests and providing simple UI based on Material design. Release version of architect-api is currently available on Pypi, to install it, simply execute:
pip install architect-api
To bootstrap latest development version into local virtualenv, run following commands:
virtualenv -p python3 venv
source venv/bin/activate
git clone git@github.com:cznewt/architect-api.git
cd architect-api
python setup.py install
Or you you can install service by pip architect-api
package.
virtualenv venv
source venv/bin/activate
pip install architect-api -e
Initial Setup for UI¶
Architect-api uses the npm
to install its JavaScript dependencies, which
are collected by django-npm
static files collector. You can install all
static nodesj libraries by following commands. More about installing Node.js
and NPM can be found at https://www.npmjs.com/get-npm.
apt install npm
cd architect-api
npm install
Architect uses the Bootstrap 4 library wich uses SASS 3.5 style preprocessing. No python SASS interpreter does it well so we need to get the ruby’s gems this time. The static file compress utility uses this ruby binary to perform the processing of SASS styles. You can install the SASS compiler by following commands. More about installing SASS can be found at http://sass-lang.com/install.
apt install gem ruby-dev
sudo gem install sass --no-user-install
sass --version
The saas --version
command should return Sass 3.5 or higher.
Now you can collect all your static assets, run following command in architect base dir and sourced.
$ python manage.py collectstatic --noinput
X static files copied to '/python-apps/architect/static', Y unmodified.
Now you can compress and compile your static assets, in architect base dir and sourced run following command.
$ python manage.py compress
Found 'compress' tags in:
/python-apps/architect/architect/templates/_head.html
/python-apps/architect/architect/templates/_body.html
Initial Setup for Database¶
You must synchronise your database content with the current migration scheme, command will create entire schema and apply all the migrations if run for the first time. In architect base dir and sourced run following command.
python manage.py migrate
You need also setup your user credentials if creating a new deployment.
python manage.py createsuperuser
You can install sample metadata fixtures by following command.
$ python manage.py loaddata sample_saltstack
Installed 614 object(s) from 2 fixture(s)
You must set database configuration by settings in architect-api configuration file. Example PostgreSQL settings in architect-api configuration file.
databases:
default:
ENGINE: django.db.backends.postgresql_psycopg2
NAME: architect
USER: architect
PASSWORD: password
HOST: 127.0.0.1
PORT: 5432
The similar applies for the cache backend, which can be changed to the Memcached backend, for example:
caches:
default:
BACKEND: django.core.cache.backends.memcached.MemcachedCache
LOCATION: 127.0.0.1:11211
Main Configuration File¶
You provide one YAML configuration file for all settings. The default
location is /etc/architect/api.yml
.
You can setup basic configuration of database and cache also you can provide defaults for your initial inventories, managers and monitors.
You can override the default location of the configuration file by setting the
ARCHITECT_CONFIG_FILE
environmental variable to your custom location.
The configuration file currently supports following options:
databases:
default:
ENGINE: django.db.backends.postgresql_psycopg2
...
caches:
default:
BACKEND: django.core.cache.backends.memcached.MemcachedCache
...
monitor:
monitor01:
name: Dashboard 01
...
manager:
manager01:
engine: salt
...
inventory:
inventory01:
engine: reclass
...
The databases
and caches
keys are used in the application settings.
But the monitor
, manager
and inventory
configuration settings need
to be sychronised to database by management commands in architect base dir and
sourced.
$ python manage.py sync_inventories
Inventory "inventory01" resource updated
...
$ python manage.py sync_managers
Manager "manager01" resource updated
...
$ python manage.py sync_monitors
Monitor "monitor01" resource updated
...
You can run the configuration multiple times and update existing resources. The actual resources used are stored in the database and can be changed at the architect’s admin app available at http://127.0.0.1:8181/admin/ after you start the development server.
Look at the the documentation pages for individual inventory, manager or monitor configuration options and installation problems.
Running Development Server¶
To start development server, in architect base dir and sourced run following command.
$ python manage.py runserver 0.0.0.0:8181
Performing system checks...
System check identified no issues (0 silenced).
January 27, 2018 - 13:12:47
Django version 2.0.1, using settings 'architect.settings'
Starting development server at http://0.0.0.0:8181/
Quit the server with CONTROL-C.
Service architect-worker
Installation¶
The architect relies on standalone workers to perform the tasks asynchronously. For the development environment, you can just simply install redis server to serve as message bus by following command.
apt install redis server
Now you can start running your architect worker instances. The redis is hardcoded and celery can be replaced by airflow, this is up to discussion.
Running development worker¶
To start development worker, in architect base dir and sourced run following command.
$ celery -A architect worker -l info
-------------- celery@wst01 v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.10.0-42
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: architect:0x7ff566a38e80
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. architect.celery.debug_task
. get_manager_status_task
[2018-01-27 13:15:55,852: INFO/MainProcess] Connected to redis://localhost:6379//
[2018-01-27 13:15:55,860: INFO/MainProcess] mingle: searching for neighbors
[2018-01-27 13:15:56,880: INFO/MainProcess] mingle: all alone
[2018-01-27 13:15:56,892: INFO/MainProcess] celery@<your-node-hostname> ready.
You should see celery@<your-node-hostname> ready
in the output of the
command run. If not, check if redis service systemctl status redis-server
is running. You need at least one instance of worker running.
Service architect-client
Installation¶
Following steps show how to deploy and configure Architect Client. You need to install client on configuration management servers to integrate the inventory service.
pip install architect-client
Create configuration file /etc/architect/client.yml
for client.
project: project-name
host: architect-api
port: 8181
username: salt
password: password
Complete Installation Scripts¶
Architect API Dependencies¶
You can install requried services.
#!/bin/bash -ex
export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8
printf "Update Ubuntu ..."
sudo apt-get update -y
printf "Install servers ..."
apt-get -y install memcached redis-server postgresql-9.5
printf "Setup Postgresql ..."
sudo -u postgres psql -c "CREATE DATABASE architect"
sudo -u postgres psql -c "CREATE USER architect WITH PASSWORD 'password'"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE architect TO architect"
Or use local docker-compose.yml
to start the same services.
Architect API Core¶
Install full development environment.
#!/bin/bash -ex
export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8
printf "Update Ubuntu ..."
sudo apt-get update -y
printf "Installing Python 3 dependencies..."
apt-get -y install python-virtualenv python3-dev python3-pip libxml2-dev libxslt1-dev libffi-dev graphviz libpq-dev libssl-dev
printf "Upgrading pip"
pip3 install --upgrade pip
mkdir -p /etc/architect
cat << EOF > /etc/architect/api.yml
databases:
default:
ENGINE: django.db.backends.postgresql_psycopg2
NAME: architect
USER: architect
PASSWORD: password
HOST: 127.0.0.1
PORT: 5432
caches:
default:
BACKEND: django.core.cache.backends.memcached.MemcachedCache
LOCATION: 127.0.0.1:11211
inventory:
sample-cluster:
engine: hier-cluster
service_class_dir:
system_class_dir:
cluster_class_dir:
class_dir: /srv/architect/mcp/classes
formula_dir: /srv/salt-formulas/formulas
sample-deploy:
engine: hier-deploy
class_dir: /srv/architect/mcp/classes
node_dir: /srv/architect/mcp/nodes/sample-deploy
EOF
git clone https://github.com/cznewt/architect-api.git /opt/architect
cd /opt/architect
printf "Installing architect-api code"
virtualenv -p python3 venv
. venv/bin/activate
pip install -r requirements/base.txt
pip install psycopg2-binary
pip install git+https://github.com/salt-formulas/reclass.git@python3
printf "Installing static files tools"
apt-get -y install npm rubygems ruby-dev
sudo gem install sass --no-user-install
npm install
python manage.py collectstatic --noinput
python manage.py compress
python manage.py migrate
# Now you can run following 2 services
# python manage.py runserver 0.0.0.0:8181
# celery -A architect worker -l info
Repository Image Builders¶
#!/bin/bash -ex
export DEBIAN_FRONTEND=noninteractive
export LC_ALL=en_US.utf8
printf "Update Ubuntu ..."
sudo apt-get update -y
printf "Install Raspberry Pi build dependencies ..."
apt-get install -y debootstrap debian-archive-keyring qemu-user-static binfmt-support dosfstools bmap-tools whois bc crossbuild-essential-armhf
printf "Install BeagleBone build dependencies ..."
apt-get install -y m4 bmap-tools dosfstools rsync git-core kpartx wget parted pv
Sample Hierarchical Inventory¶
#!/bin/bash -ex
mkdir -p /srv/architect/mcp/classes/deployment
mkdir -p /srv/architect/mcp/classes/service
mkdir -p /srv/architect/mcp/classes/cluster/sample/infra
mkdir -p /srv/architect/mcp/nodes/sample-deploy
if [ ! -d "/srv/architect/mcp/classes/system" ]; then
git clone https://github.com/Mirantis/reclass-system-salt-model /srv/architect/mcp/classes/system
fi;
if [ ! -d "/srv/salt-formulas" ]; then
git clone https://github.com/salt-formulas/salt-formulas.git --recursive /srv/salt-formulas
fi;
for i in /srv/salt-formulas/formulas/* ; do
if [ -d "$i" ]; then
service=$(basename "$i")
formula="${service/-/_}"
if [ -d "/srv/salt-formulas/formulas/$service/metadata/service" ]; then
if [ ! -L "/srv/architect/mcp/classes/service/$formula" ]; then
ln -s "/srv/salt-formulas/formulas/$service/metadata/service" "/srv/architect/mcp/classes/service/$formula"
fi;
fi;
fi;
done
cat << EOF > /srv/architect/mcp/classes/cluster/sample/infra/config.yml
classes:
- service.git.client
- system.linux.system.single
- system.linux.system.repo.mcp.salt
- system.salt.master.pkg
parameters:
_param:
salt_master_host: 127.0.0.1
salt_master_environment_name: dev
salt_master_environment_repository: https://github.com/salt-formulas
salt_master_environment_revision: master
salt_minion_ca_authority: salt_master_ca
salt:
master:
user:
salt:
permissions:
- '.*'
- '@local'
- '@wheel' # to allow access to all wheel modules
- '@runner' # to allow access to all runner modules
- '@jobs' # to allow access to the jobs runner and/or wheel modules
engine:
architect:
engine: architect
project: sample-deploy
host: 127.0.0.1
port: 8181
username: architect
password: password
EOF
cat << EOF > /srv/architect/mcp/classes/deployment/sample-deploy.yml
parameters:
_param:
cluster_name: sample-deploy
cluster_domain: sample.deploy
EOF
cat << EOF > /srv/architect/mcp/nodes/sample-deploy/cfg01.sample.deploy.yml
classes:
- cluster.sample.infra.config
- deployment.sample-deploy
parameters:
_param:
linux_system_codename: xenial
linux:
system:
name: cfg01
domain: sample.deploy
EOF