Web gateway¶
Description¶
Installs services needed for a public facing web server.
Components¶
nginx¶
Interaction¶
Service users may invoke sudo /etc/init.d/nginx {reload,restart} to activate updated nginx configurations.
Configuration¶
Virtual hosts¶
We provide a basic nginx configuration, but do not define virtual hosts. Service
users may create individual config files in /etc/nginx/local
that will
be included in a http
context.
The file /etc/nginx/local/example-configuration
may serve as a start.
Copy this file to something like /etc/nginx/local/vhost.conf
and edit it. You must at least:
replace www.example.com with the action virtual host name
adapt the proxy statements to match your application.
Anonymization¶
If you want IP addresses (IPv4 and IPv6) in nginx log files to be anonymized, use the log format “anonymized” by adding the following line inside your server declaration block:
access_log /var/log/nginx/${server_name}_access.log anonymized;
This way, only a /24 prefix for IPv4 addresses and a /48 prefix for IPv6 addresses will be logged.
nginx’ error log in /var/log/nginx/error.log
can not be anonymized this
way. You can, however, modify it’s log level (or turn off the error log
completely) by overriding the error_log directive
outside
your server declaration block.
Performance logging¶
In addition to the normal access log files found in
/var/log/nginx/site.access_log
(common log format), nginx logs
performance data for each request. There is a single log file
/var/log/nginx/performance.log
which has lines consisting of the
following fields:
request time in ISO 8601 format
request identifier (see Request IDs below)
HTTP method
full URL in the form “SCHEME://HOST/URI*”
request status
length (bytes) of the response sent to the client
length (bytes) of the request received from the client
p if the request was pipelined, - otherwise
total request time (from receiving the first byte until sending the last byte)
comma-separated list of upstream request processing times (several items may be logged in case of upstream retries)
mod_gzip compression ratio or - if no compression took place.
Example log line:
2015-04-17T16:56:36+02:00 32990.1 GET "http://flyingcircus.io/@@/gocept.flyingcircus/js/jquery.min.js" 200 38667 922 . 0.8 "0.002, 0.006" 2.44
Example Pandas data import:
import pandas
performance = pandas.read_csv(
'performance.log', sep=' ', quotechar='"', parse_dates=['timestamp'],
na_values='-', names=('timestamp', 'id',
'method', 'url', 'status', 'response_length', 'request_length', 'pipe',
'request_time', 'upstream_times', 'gzip_ratio'))
Example PostgreSQL data import:
CREATE TABLE performance (
timestamp TIMESTAMP WITH TIME ZONE,
id VARCHAR(24),
method VARCHAR(10),
url TEXT,
status INT,
response_length INT,
request_length INT,
pipe CHAR(1),
request_time FLOAT,
upstream_times TEXT,
gzip_ratio FLOAT
);
COPY performance FROM 'performance.log' (FORMAT CSV, DELIMITER ' ',
QUOTE '"', NULL '-');
Request IDs¶
nginx
assigns each HTTP request a unique id. In the default configuration,
the id gets logged to performance.log
and passed to upstream servers in
the X-Nginx-Id
request header.
See /etc/nginx/nginx.conf
for details. Users who configure their own
proxy_set_header
directives should be aware that they have to set this header
on their own if needed.
Note
Request ids are not guaranteed to be unique forever. It is reasonable to assume that request ids will not repeat on the same host on the same day.
Monitoring¶
By default, the presence of nginx processes is checked.
To get checks for individual web pages (presence, specific content, response time), please notify the support.
Testing¶
To test your settings, e.g. to ensure you did not introduce some syntax error, you can use sudo /etc/init.d/nginx configtest.
mod_security¶
The open-source web application firewall mod_security is included on all web gateways. In the default configuration, however, it is inactive.
Configuration¶
To activate mod_security
, enable it in a nginx configuration section
(e.g., server
or location
):
ModSecurityEnabled on;
ModSecurityConfig /etc/nginx/modsecurity/modsecurity.conf;
This system-wide configuration file contains some useful defaults like
activating the base rule set etc. Initially, mod_security
runs in report-only
mode. Additional configuration like switching to policing mode or including
custom rule sets goes into /etc/nginx/modsecurity/local.conf
. Service
users can also put completely different mod_security configuration files into
the /etc/nginx/modsecurity
directory and use these in place of the
default configuration file.
Log files¶
All mod_security
events are logged into a directory hierarchy under
/var/log/nginx/modsec_audit
. Each event is written to a new file.
Log directories which are more than a day old are archived as tarballs under
/var/log/nginx/modsec_audit_DATE.tar.xz
and are subject to normal log
retention.
AWStats¶
The system generates awstats configuration files automatically for every vhost
configured in nginx. awstats configuration generally goes into
/etc/awstats/vhosts.d
and individual configuration files follow the
naming convention awstats.VHOST.conf
.
awstats configuration for individual vhosts can be augmented with custom
configuration. Just put it into /etc/awstats/vhosts.d/local.VHOST.conf
and it will be picked up during the next awstats run.
Application deployments may install configuration files for additional vhosts or
replace the system-generated awstats configuration altogether. To do this,
remove the system-generated configuration files (which belong to the awstats
user) and replace them with files owned by an service user. Such files will not
be touched by the automatic configuration update.
Web statistics generated on a VM can be accessed through the My Flying Circus dashboard.
By default, a list of all configured vhosts is presented. The presentation can
be adjusted by creating /etc/awstats/pages.cfg
. See the comments in
/srv/www/localhost/cgi-bin/stats.cgi
for details.