Deploy and develop Poio

Poio Website

The repository is here:

https://github.com/cidles/poio-site

This will install all requirements and prepare the server (Flask webapp) for launch. The Poio website used Buildout to install all the necessary packages. The Poio website comes with a bootstrap script, so you don’t have to install anything except Python.

Initialize your enviroment

Start by bootstraping the buildout environment:

$ python bootstrap.py

Next you have to install all dependencies using the buildout script:

$ sudo bin/buildout

Get language data from server

You have to download all the language data from our Amazon server (this may take a while):

$ python get_corpus_data.py

Run the tests

Before starting the server you should run our tests to ensure that everything is working properly:

$ bin/test

Start the server in development mode

Finally you can start the server:

$ bin/flask-ctl debug fg

Example deployment on Ubuntu

For deployment we use gunicorn, supervisor and nginx. First, install nginx and supervisor via apt-get:

# apt-get install nginx
# apt-get install supervisor

The example configuration that comes with the poio-site project and the following steps assume that the code of the project is in the directory /home/sites/poio. Please change this path in all configurations options if you installed the code somewhere else.

Gunicorn is already installed in the buildout environment if you followed the instruction in Initialize your enviroment. An example gunicorn config file gunicorn.conf.py is in the main folder of the poio-site project. You may try to start the gunicorn server manually:

$ bin/gunicorn main:app -c gunicorn.conf.py -p gunicorn.pid

The supervisor daemon is then responsible to start the gunicorn server and keep it running, i.e. restarting it whenever it stopped or crashed. For the Poio website we need a supervisor configuration script, which we store in /etc/supervisor/conf.d/poio.conf. The content is:

[group:poio]
programs=poio

[program:poio]
command=/home/sites/poio/bin/gunicorn main:app -c gunicorn.conf.py -p gunicorn.pid
directory=/home/sites/poio
autostart=true
autorestart=true
redirect_stderr=true

You now have to tell supervisor that we have a new application. This will also automatically start the gunicorn process:

# supervisorctl update

Restarting the website is also done via supervisorctl:

# supervisorctl restart poio

The last step is now to tell nginx tp proxy all requests to the gunicorn process. This will make the website available to the public. An example configuration file for nginx is:

upstream poio {
  server 127.0.0.1:8200;
}

server {
  server_name             yourdomain.com;
  listen                  80;
  keepalive_timeout       70;

  access_log              /var/log/nginx/access-poio.eu.log;
  error_log               /var/log/nginx/error-poio.eu.log;

  location / {
    proxy_redirect      off;
    proxy_set_header    Host                    $host;
    proxy_pass          http://poio;
  }
}

We store the configuration file in /etc/nginx/sites-available/www.poio.eu. To activate the site you still have to create a symbolic link of the file in the directory /etc/nginx/sites-enabled:

# ln -s /etc/nginx/sites-available/www.poio.eu /etc/nginx/sites-enabled/

Then restart nginx:

# /etc/init.d/nginx restart

That’s it. You should now have the Poio website deployed and running on your server.

Develop with PyCharm

  • Start by creating a new project with the following settings:
    • Project name: Poio Site
    • Location: <PATH_TO>/poio-site/src/main/
    • Project type: Flask Project
    • Interpreter: Python 2.7
  • After you press Ok PyCharm will prompt if you want to create a project from existing sources, press Yes.
  • In order to run the server from PyCharm you need to add a new confguration for the server, to do this:
    • On the menu bar go to Run and open Edit Configurations...;
    • Press the + sign and from the dropdown menu choose Python.
  • Fill in the new confifuration with the following settings and press Ok:
    • Name: Poio Site Server
    • Script: bin/flask-ctl
    • Script parameters: debug fg
    • Working directory: <PATH_TO>/poio-site/

Now every time you want to start the server make sure that the selected configuration on the menu bar is Poio Site Server and just press Run (play button).

Poio Corpus

The repository is here:

https://github.com/cidles/poio-corpus

Install the requirements for Poio Corpus

You can find a list of all requirements in REQUIREMENTS.txt.

This documentation is for Linux/Ubuntu only.

You need to install the following packages with apt-get:

  • python-lxml
  • python-numpy
  • python-scipy

For example:

$ sudo apt-get install python-lxml

You need to install the following packages with easy_install or pip:

  • requests
  • beautifulsoup
  • graf-python
  • poio-api
  • rdflib
  • cython
  • sparsesvd
  • regex
  • s3cmd

For example:

$ sudo easy_install requests

In addition, some packages must be downloaded from github:

To install the packages run:

$ sudo python setup.py install