15:02:35 #startmeeting OpenStack 15:02:35 Meeting started Mon May 11 15:02:35 2015 UTC. The chair is msuchy. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:35 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:02:52 Hi, 15:02:53 I have prepared the writings in advance therefore I will go in quite fast peace. Do not hesitate to interrupt me and ask questions immediatelly. 15:03:04 As you may know we have old Fedora Cloud. It is based on OpenStack (OS) Folksom. So pretty old. Installed manually without any chance to install it quickly again if there is some problem. 15:03:33 Our new Fedora Cloud is based on OS Icehouse and installed using Ansible. Therefore in case of some fatal failure we should be able to rebuild whole OS in matter of hours. 15:03:41 Our new Fedora Cloud is based on OS Icehouse and installed using Ansible. ThWhy Icehouse and not Juno? This whole initiate started year ago. When we started even with Havana. But then switched to Icehouse. When Juno was released I tried to switch to Juno, but there was non-trivial amount of bugs, which worked in Icehouse so I reverted. There was no time to be hero. I filled huge amount of bugs reports anyway.erefore in case of some 15:03:43 fatal failure we should be able to rebuild whole OS in matter of hours. 15:04:06 BTW big thanks to larsks who helped me we several issues. 15:04:15 The new cloud have controller on fed-cloud09 with webUI interface on https://fed-cloud09.cloud.fedoraproject.org/dashboard/ (mind the "s" in protocol, there is no automatic redirect). 15:04:25 Controller have installed all services (compute, network, image...), there are six other machines which acts as compute nodes (fed-cloud10..fed-cloud15). 15:04:26 larsks++ Lots of great help with openstack. 15:04:26 nirik: Karma for larsks changed to 1: https://badges.fedoraproject.org/badge/macaron-cookie-i 15:04:39 Current installation works. More or less. Last time it still required one human intervention. But otherwise it is fully automatic. There is still a lots of places for improvements though. 15:04:49 The new OS have in total: 288 VCPUs, 686 GB RAM and 20 TB on DellEqualogc for persistent volumes. HDD on nodes are used for persistent volumes (300GB on controller), for Swift storage (100GB on each node - but more on this later) and for rootFS and swap of VM (3TB on each compute node). 15:04:59 Note: there is no separate storage for Glance as it is configured to use Swift as backend (it is stored in container named "Glance" under "glance" users - not normaly visible). 15:05:17 I will start about how to migrate current VM from old OS to new OS. 15:05:26 With old OS we used mainly EC2 API. This is not recommended and it really introduce issues. So we know use native OS API. 15:05:34 Previously to spin up VM and attach volumes we used ./tasks/persistent_cloud.yml, now we have ./tasks/persistent_cloud_new.yml (all path are relative to ansible.git root). 15:05:45 Example how to use is are: 15:05:56 inventory/host_vars/copr-be-dev.cloud.fedoraproject.org 15:06:03 playbooks/groups/copr-backend-newcloud.yml 15:06:11 Mind that most time you need to use UUID to identify resources so comments for humans are welcome in playbook. 15:06:18 In filter_plugins/openstack.py there are some filter to change ID to name and vice versa. But e.g. filters could not be used in vars files, so the usage is quite limited. 15:06:26 Due architecture of Copr we have two shared networks copr-net, and copr-dev-net. They are used for Copr builders and they are shared so Copr backend (from persistent tenant) can comunicate with builders. But due this setup, you always have more then one network available when spining up VM, so no network is automatically assigned to VM and you always have to specify network to be used. 15:06:44 In previous OS instance you automatically get public IP to every instance (sometimes two, due some buggy behaviour). This is no longer true and you no longer get public IP. You must explicitly assign it from floating IP pool. And that one is quite limited (38 IPs). We can get more IPs once old OS is decomissioned. 15:06:55 Now about installing: 15:07:07 At the outset - the idea is the if you create new user, install new cloud image, create new tenant, security group... you will do that in controller node playbook and *not* manually. So in case of total failure we can run this playbook again and restore all data/service as quickly as possible. 15:07:08 Keep this in mind. This is biggest change from old OS instance. 15:07:19 Most important variables are in vars/fedora-cloud.yml and sensitive variables are in /srv/private/ansible/files/openstack/passwords.yml 15:07:20 There are some files in files/fedora-cloud/ 15:07:36 And you can read some documentation in our infra-docs.git in cloud.txt 15:07:46 The playbook for controler node is playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml 15:07:54 The playbook for compute nodes is in playbooks/groups/openstack-compute-nodes.yml and is based on role, which is defined in roles/cloud_compute/tasks/main.yml. 15:08:04 I will start with this one one as it is quite easy. It just configure ephemeral and swift storage and set up neutron-openvswitch-agent and nova according upstream OS documentation and that is all. 15:08:04 The controller playbook should be runable anytime and correctly just add new images/tenats/security groups right? 15:08:41 nirik: mostly yes. I will talk about the condition little bit later 15:08:46 * nirik nods ok. 15:08:53 The only drawback is -- if you install Compute nodes and after that you install Controller (for whatever reason), you *must* restart neutron-openvswitch-agent and openstack-nova-compute services (it is commented out at the very end of the playbook). 15:09:05 The controller playbook is quite something different and far more complicated (again see playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml). 15:09:06 Before I dive into this: 1) I'm aware that most task does not have "name" defined -- feel free to do that. I focused so far on the functionality. 15:09:08 2) Right now, most services are always restarted. I know that I can use "notify", however sometimes it is tricky and it is not clear which service(s) should be restarted. So again - I just focused on having it done. Feel free to fix that, but I have to warn you, that I tried that in past and experienced some surprises. 15:09:15 The controller playbook is quite something different and far more complicated (again see playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml). 15:09:16 Before I dive into this: 1) I'm aware that most task does not have "name" defined -- feel free to do that. I focused so far on the functionality. 15:09:18 2) Right now, most services are always restarted. I know that I can use "notify", however sometimes it is tricky and it is not clear which service(s) should be restarted. So again - I just focused on having it done. Feel free to fix that, but I have to warn you, that I tried that in past and experienced some surprises. 15:09:21 3) Swift configuration is not finished. We have 100 GB on controller node, which are used by Swift. That works. Then we have have reserved 100 GB LV on each compute node for Swift, but Swift service is not yet configured to use it. This is something which I would like to address to use soon (i.e. within few months) 15:09:38 * msuchy is sorry for sometimes duplicating lines 15:10:31 So looking at playbook, on lines 0-22 there is created storage for Swift on Compute nodes. 15:10:32 I set up ssh keys so packstack can log into compute nodes (line 64-68). 15:10:39 There is disabled SELinux as I was not brave enough to enable it, I reported too much bugs anyway. (line 80) 15:10:40 Then follow prerequisites (lines 83-104). 15:10:55 Lines 127-156 configure networking. Notice that these lines (and some other) have: when: packstack_sucessfully_finished.stat.exists == False 15:10:56 So it is executed only when packstack was never run or did not finished. When it successfully finishes then file /etc/packstack_sucessfully_finished is touched and some tasks are skipped. 15:11:11 So this initial networking looks like: eth0 is external ethernet network, eth1 is ethernet network (physicaly routed to separate switch where goes eth1 of compute nodes too). Both interfaces should be up, and if there is some leftover (br-tun) from packstack, just take it down. 15:11:12 Then we install packages (lines 161). Some of them would be install as requirements by packstack, but we need them to have before, because it creates users and we need to chown some files we are installing before packstack is executed. 15:11:14 E.g. SSL certificates (lines 179-203). 15:11:21 Then MariaDB is installed (line 206). While packstack can do that, I wanted to be sure that it is secured and testdb and anonymous users are removed. 15:11:36 Workaround follows (lines 239-257), reported to OS team, but not fixed when I was writing this playbook. This can be removed later in future. 15:11:38 Very important part is line 260. It execute packstack with answer file (stored in fedora-cloud/packstack-controller-answers.txt). 15:11:39 Just this one step run aproximately 20 minutes. However it does enourmous work. Originally I wanted to everything according upstream OS wiki. But that playbook was huge, and after 2 weeks I was even in quater of that documentation. 15:12:01 Packstack is designed to configure your OS instance, and if you want to change something, you will alter your answer file and run it again. In the heart is is just python code, which generate Puppet manifests, which are executed. Howevere this is just theory and puppet have its limit. E.g. it could not configure SSL for services, it could not configure Equalogic... 15:12:13 So I used it just for initial configuration. Only when you start with bare metal. Once it finish, it touch file "/etc/packstack_sucessfully_finished" and is *not executed anymore*. Therefore if you alter answer file and execute the playbook now, it will have no effect. Just keep that in mind. 15:12:31 nirik: ^^^ this should answer your question 15:14:02 nirik: so very base configuration done by packstack could not be altered (like which protocol is used for tunneling), but tenants, sec-group, quotas, sub-networks etc. can and should be altered in playbook 15:14:19 We continue (lines 265-273) with setting up ssh keys for nova user, this is used for Cold migration of VM. 15:14:30 Lines 282-352 flip endpoints from HTTP to HTTPS. The port stay the same, just protocol is changed. This is just copy'n'paste but keystone, which is tricky as you could not use keystone to change keystone endpoint and it use admin token. 15:14:43 Technicaly the SSL comunication is handled by HAProxy and all services are altered to listen on nonstandard port and haproxy tunnel it. (lines 359-449 and file files/fedora-cloud/haproxy.cfg). 15:14:53 On lines 381-385 there is as side effect set Swift as backend storage for Glance. All images are stored in Swift container named "glance" under "glance" user. 15:15:18 Then I set up Cinder. The LVM part is set up by packstack. It is "just" 300 GB and we can rather use it for something else as EquaLogic have 20 TB, but packstack needs either LVM, NFS or (?) Gluster and reject to proceed if something is not given. So I use that LVM. And the EquaLogic is configured (lines 469-486). 15:15:26 I created two types, so you can actually choose where your volume will be created (lines 489-495). 15:15:37 On lines 509 I create non-standard flavour. m1.builder is for Copr. ms.* are the same as m.* but with swap and then I created some flavor inspired by AWS. 15:15:55 On line 530 I upload general Cloud Images. Then follow RHEL images, which have to be downloaded manually. In comments there is HOWTO. 15:16:08 Then I create tenants. 15:16:10 Then I create users (line 635). The passwords are stored in /srv/private/ansible/files/openstack/passwords.yml and are used so we can upload your SSH key under your account (line 629). The keypair is taken from FAS. Once you change your password, and if you alter your ssh key, you have manage it yourself. 15:16:22 Kevin, Patrick and me can see all tenants (line 663) so we do not have to login as admin so often. 15:16:32 Then I create external network (line 717) 15:16:41 And network for every tenants. (line 759-822). Please try to keep the comments up to date, when creating new tenant. 15:16:56 Copr-net and coprdev-net are created as shared, because Copr backend machine have two NIC, one with persistent net (for external communication), and second with copr net (to communicate with builders). 15:18:07 BTW I forgot to specify that the manual part which still reside in this set up is somewhere aftere running packstack (initially) and we had to always reboot that machine. I'm still not sure where and why, but it always fixed the networking problem. 15:18:20 cont. with playbook: 15:18:28 Then I create security groups. Due bug in ansible module I have to name them with - suffix. E.g ssh-anywhere-persistent, ssh-anywhere-scratch. 15:18:35 BTW I already persuade that Adam Samalik will fix those modules and create bunch of new as his bachelor thesis. So in future we will hopefully replace all shell commands by ansible modules. 15:18:49 And at the very bottom is definition of quota. Right now just for Copr. If you want to change quota for some tenant feel free to put it here according this template. 15:19:02 Beside those services I mentioned, there is installed OS Firewall. The difference between OS Firewall and Security Groups are that SC are managed by tenants admin and visible for them. While OS Firewall is managed by admin only and not visible to users. I did not set any rule there yet. 15:19:11 Another service is Ceilometer which enable you to display various meassurements. E.g. Network traffic for each tenant or VM. Accumulated CPU time etc. It can even set up various tresholds and alerts but if the treshold is reached nothing happen. You can only query the DB if the treshold was reached. It can be likely used by Nagios, but that is beyond my knowledge. 15:19:21 And I installed LoadBalancer as it was just one line in answer file, but no pool is defined yet. 15:19:37 Last note - to controller node you ssh as root (it does not have fas_client role). To compute nodes you ssh under your FAS accound and then call sudo with your token and OTP password. 15:19:50 That is all I wanted to share with you. 15:19:57 It would be nice if you can start working on migrating VM from old OS instance to the new one. Once old OS is empty, we can move some HW to new one as compute nodes. And keep two machines as playground for testing deployment of next-gen OS instance OS-Kilo or something later. As the upgrades of OpenStack are not supported. 15:20:16 Question? Comments? 15:20:40 * msuchy will give you time to catch up with reading and checking those playbooks in our git 15:20:56 is there a timetable for migrating production copr? 15:21:12 we should also add monitoring for the new cloud 15:21:33 (at least controller and compute nodes) 15:22:30 nope. I started migrating that big volume on Wednesday. And it is still not finished. :) It sometimes crash with packet error, which according the mr. Google is due insufficient memory for rsync on remote machine. 15:23:05 ok. Also, in new cloud can we give copr more builders? ;) I know the current setup has been limited and sometimes there's quite a backlog. 15:23:05 I put rsync in for i in `seq 10` but it was obviously to small number as it did not copy everything over weekend 15:23:20 in the old cloud different instances were unable to talk to each other (for example copr to jenkins, or koschei to copr), is this going to be fixed in the new cloud? 15:24:30 So on copr migration, the intent is to copy all data, stop old copr, sync again, then change dns to point to new ips? or were you going to try and take down the ip's on the old instances and add them to new so no dns change? 15:24:39 nirik: yes, that is plan. During testing we found bug in OS when we reached level of 200 builders (you could not use CLI tools then), but yes the limit will be higher compared to old one 15:25:38 mizdebsk: you can communicate using public (floating) IP. you can not using internal. Unless one of the networks are share (this is example of copr) 15:26:19 right, things in the same tenant should be able to talk to each other, but not seperate ones. 15:26:31 mizdebsk: that is limitation of OS itself, not limitation of our instance 15:26:45 i see 15:26:52 does the new cloud have any non-x86 machines? 15:27:00 mizdebsk: nope 15:27:13 well, it's a design decision. You don''t want account a being able to access account b's resources... 15:27:22 and AFAIK it is not supported in OS Juno (last one released) 15:28:25 as far as migration of other stuff, I think we can setup a wiki page and contact people with their initial password and ask them to migrate, then set a date we will be shutting down the old cloud. 15:29:20 * msuchy nod 15:29:32 any plans on deploying ceilometer for metering/monitoring? 15:29:56 dotEast2015: it is already deployed 15:29:57 I think msuchy said that ceilo was installed 15:31:05 ah, I see. msuchy, could just highlight the issue of cross tenant communication 15:31:42 dotEast2015: however I think that only admin have access to that. if you would like to see it we would have to use some role and somehow set it up. not sure how, right now though 15:34:38 hmm, I see. I guess one might be given member level access to the tenant and see if that does it 15:34:47 msuchy: was there a sop or doc that mentioned about ssl certs? also did we want to try and get a more official cert? or not bother? 15:36:23 nirik: I'm not aware of such SOP. I resolved all my problem by making the cert localy trusted 15:36:36 in playbooks/hosts/fed-cloud09.cloud.fedoraproject.org.yml line 185 15:37:06 nirik: so official cert would be nice, but not neccessery 15:37:23 right, but for others using it... 15:38:16 yes 15:38:20 we can make a short note when we tell people to migrate I guess. 15:39:47 The playbooks are not currently 100% idempotent. The compute nodes would be but for an ansible bug, the controller might need a slight bit of tweaking. 15:40:59 * nirik can work on that and/or provide info for others. ;) 15:41:14 anyhow, no further questions. Thanks much for all your work on this msuchy. :) 15:41:24 nod. however the compute nodes are not due ansible bug, but due OS. When you reinstall controller (e.g run packstack) you have to restart service nodes on compute node. I do not why, but that is reality of OS. 15:41:43 well, ansible complains about: 15:42:02 fed-cloud10.cloud.fedoraproject.org May 11 2015 04:19:56 83 CHECK_DIFF:CHANGED yum state=present name=https://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-4.noarch.rpm task_userid:root 15:42:10 it is present tho. 15:42:17 it just complains because it's a url I guess. 15:43:17 nirik: aha, we can replace it with download and yum install that file. 15:43:46 msuchy++ for the new cloud :) 15:43:46 pingou: Karma for msuchy changed to 2: https://badges.fedoraproject.org/badge/macaron-cookie-i 15:44:02 right and don't run the download/yum install if /etc/yum.repos.d/whatever.repo exists. 15:44:23 thank you for listening. You can contact me any time about our OS set up and how/why is something done. 15:44:46 #endmeeting