Gfilui

Deploying openstack-base bundle. Charm deployment issues with keystone.
charm : cs:keystone-280

Problem is that charm performs config-changed hook then becomes blocked by services not running, memcached apache2, ports not running 5000 35357. I confirmed ports were not running. Both apache2 and memcached were "marked" in systemctl. I had to then unmark them in order to start them, which then allowed the keystone charm to complete deployment.

Is there a bug in the charm's hooks preventing it from unmarking the other services?

user@us-a-reg-maas:~/openstack-base$ juju --version
2.3.7-xenial-amd64
root@ilo-7432:/var/log/apache2# apache2 -V
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2018-04-18T14:53:04
Server's Module Magic Number: 20120211:52
Server loaded: APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture: 64-bit
root@ilo-7432:/var/log/apache2# memcached -V
memcached 1.4.25
root@ilo-7432:/var/log/apache2# keystone-manage --version
13.0.0

keystone ports not running: 5000 35357
attempted to start apache2

root@ilo-7432:/var/log/apache2# apache2 --help
[Thu May 24 14:53:03.640936 2018] [core:warn] [pid 4982] AH00111: Config variable $APACHE_LOCK_DIR is not defined
[Thu May 24 14:53:03.640986 2018] [core:warn] [pid 4982] AH00111: Config variable $APACHE_PID_FILE is not defined
[Thu May 24 14:53:03.640998 2018] [core:warn] [pid 4982] AH00111: Config variable $APACHE_RUN_USER is not defined
[Thu May 24 14:53:03.641001 2018] [core:warn] [pid 4982] AH00111: Config variable $APACHE_RUN_GROUP is not defined
[Thu May 24 14:53:03.641009 2018] [core:warn] [pid 4982] AH00111: Config variable $APACHE_LOG_DIR is not defined
[Thu May 24 14:53:03.644603 2018] [core:warn] [pid 4982:tid 140373872609152] AH00111: Config variable $APACHE_LOG_DIR is not defined
AH00526: Syntax error on line 74 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:$APACHE_LOCK_DIR
Found apache2 was masked
root@ilo-7432:/var/log/apache2# sudo systemctl unmask apache2
keystone showing haproxy and memcached not running :
root@ilo-7432:/var/log/apache2# sudo systemctl restart haproxy
Failed to restart haproxy.service: Unit haproxy.service is masked

.

I restarted haproxy

root@ilo-7432:/var/log/apache2# sudo systemctl unmask memcached
Removed symlink /etc/systemd/system/memcached.service.
root@ilo-7432:/var/log/apache2# sudo systemctl restart memcached
root@ilo-7432:/var/log/apache2# systemctl status memcached
* memcached.service - memcached daemon
 Loaded: loaded (/lib/systemd/system/memcached.service; disabled; vendor pres
 Active: active (running) since Thu 2018-05-24 15:19:59 UTC; 24s ago
 Main PID: 13617 (memcached)
 Tasks: 6
 Memory: 472.0K
 CPU: 11ms
 CGroup: /system.slice/memcached.service
 `-13617 /usr/bin/memcached -m 64 -p 11211 -u memcache -l ::1

May 24 15:19:59 ilo-7432 systemd[1]: Started memcached daemon.
root@ilo-7432:/var/log/apache2# systemctl enable memcached
Synchronizing state of memcached.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable memcached
insserv: warning: current start runlevel(s) (empty) of script `memcached' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `memcached' overrides LSB defaults (0 1 6).
root@ilo-7432:/var/log/apache2# systemctl restart memcached
jluczani@us-a-reg-maas:~/openstack-base$ juju show-status keystone
Model Controller Cloud/Region Version SLA
us-a-001 juju us-a-reg-maas 2.3.7 unsupported

App Version Status Scale Charm Store Rev OS Notes
keystone 13.0.0 active 1 keystone jujucharms 280 ubuntu exposed
mysql 5.6.37-26.21 active 1 percona-cluster jujucharms 264 ubuntu 
neutron-gateway 12.0.1 active 1 neutron-gateway jujucharms 251 ubuntu 
nova-cloud-controller 17.0.3 waiting 1 nova-cloud-controller jujucharms 309 ubuntu 
ntp 4.2.8p4+dfsg active 1 ntp jujucharms 24 ubuntu 

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0 10.76.83.102 5000/tcp Unit is ready

REVIEW juju debug-log -i keystone/0 --replay --no-tail -l INFO

. . .

unit-keystone-0: 09:34:04 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1
unit-keystone-0: 09:43:19 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1
unit-keystone-0: 09:52:34 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1
unit-keystone-0: 10:05:26 ERROR juju.worker.uniter.operation hook "identity-service-relation-changed" failed: exit status 1

user@us-a-reg-maas:~/openstack-base$ tail keystone-debug-log.log unit-keystone-0: 10:41:00 INFO unit.keystone/0.juju-log Configuring Keystone to use a random admin token.
unit-keystone-0: 10:41:00 INFO unit.keystone/0.juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
unit-keystone-0: 10:41:07 INFO unit.keystone/0.juju-log HAProxy context is incomplete, this unit has no peers.
unit-keystone-0: 10:41:07 INFO unit.keystone/0.juju-log Configuring Keystone to use a random admin token.
unit-keystone-0: 10:41:07 INFO unit.keystone/0.juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
unit-keystone-0: 10:41:08 INFO unit.keystone/0.juju-log Configuring Keystone to use a random admin token.
unit-keystone-0: 10:41:08 INFO unit.keystone/0.juju-log Loading a previously generated admin token from /var/lib/keystone/keystone.token
unit-keystone-0: 10:41:10 INFO unit.keystone/0.juju-log HAProxy context is incomplete, this unit has no peers.
unit-keystone-0: 10:41:10 INFO unit.keystone/0.juju-log Unit is ready
unit-keystone-0: 10:41:12 INFO juju.worker.uniter.operation ran "update-status" hook