18:00:02 #startmeeting Infrastructure (2016-02-11) 18:00:02 Meeting started Thu Feb 11 18:00:02 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:02 The meeting name has been set to 'infrastructure_(2016-02-11)' 18:00:02 #meetingname infrastructure 18:00:02 The meeting name has been set to 'infrastructure' 18:00:03 #topic aloha 18:00:03 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:03 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:03 #topic New folks introductions / Apprentice feedback 18:00:25 hereish 18:00:30 * pingou 18:00:36 Hello 18:00:39 .hello smdeep 18:00:39 smdeep: smdeep 'Sudeep Mukherjee' 18:01:16 * nirik takes a nice break from yelling at lvm for the meeting. :) 18:01:25 any new folks like to introduce themselves? 18:01:31 or apprentices with questions or comments? 18:01:32 nirik: very kind of you :) 18:01:49 nirik: but we wouldn't want to get in between lvm and you, if you want, one of us can run the meeting 18:01:54 :-p 18:02:21 :) I can multitask too... just takes a while between attempts due to reboot times, etc. 18:02:38 * doteast here 18:04:46 ok, no new folks or apprentices with questions? lets move on to status / info dump 18:05:05 * mirek-hm is here, but it is also putting-kids-in-bed time 18:05:17 mirek-hm: g'd luck w/ this :) 18:05:20 #topic announcements and information 18:05:20 #info Started spring cleaning with ibiblio hosts - kevin 18:05:20 #info migrated people01 to people01 - kevin 18:05:23 infra meeting? 18:05:28 cool, i actually made it for once :) 18:05:30 any other info announcements or status? 18:05:33 nb: yes, ongoing. ;) 18:05:50 nirik: ticket templating in pagure is coming in the next release :) 18:06:06 same for customized info on the PR page 18:06:17 oh nice. 18:06:36 two items from the dear-github letter checked :] 18:06:40 will templates just be initial text? or some kind of fields/enforcement? 18:06:46 intial text only 18:07:08 really similar to what we have in trac currently 18:07:09 thats fine, more than that seems like it would be pretty complex to do... 18:07:21 would need some kind of form designer 18:07:35 or rely on code provided by the users 18:07:46 which seems, un-safe 18:07:50 yeah 18:08:12 oh, one thing I thought of the other day for pagure, forgot to mention... 18:08:44 how about some kind of licensing thing... ie, you submit a PR and it asks you if you would submit under license $whatever 18:09:16 so currently for pagure.io we're fine 18:09:21 it enforces CLA :] 18:09:31 ok 18:10:05 anyhow, just a idle thought 18:10:09 shall we move on? 18:11:00 #topic Spring cleaning - kevin 18:11:21 I mentioned this on the list, but have some more info to discuss / a somewhat plain. 18:11:52 so, my plan is to reinstall the virthosts that have stg on them today (which I got stuck with virthost16 being a POS) 18:11:59 then we can reinstall stg hosts anytime after that 18:12:25 for production hosts I am going to look at bringing in a new virthost from one of the new ones we have ready for releng (but isn't in service yet) 18:12:45 migrate a bunch of the outage causing stuff to that, and then redo the prod virthosts and we can then redo the guests. 18:13:00 I'll keep the list updated with what we are at. 18:13:58 Also I will likely redo all the koji builders this weekend or something (they can be done without outage) 18:14:50 anything else on this one? 18:15:21 nirik: I'm willing to help rebuilding the guest I know best, so feel free to spread the load once the ducks are in rows :) 18:15:46 yeah, that was the plan. once virthosts are ready we can do guests in any order that makes sense. 18:15:57 Oh, one last thing... I think for databases I might try a replication thing... 18:16:07 nirik: only somewhat related, did we update the instruction on how to create a new host w/ the info about networking or is it all automagical? 18:16:12 replicate them to another host, then switch over. To avoid long downtime 18:16:43 pingou: should be all taken care of for you. ;) as long as the variables are set 18:17:05 nirik: and those are documenting? :D 18:17:23 * pingou wonders if we should have a create_new_host.py script generating the basic info :D 18:17:25 they should be the same variables we have set before... 18:17:41 cool :) 18:17:45 yeah, we have talked about that before... might be nice 18:18:08 easyfix? 18:18:40 * relrod checks in late 18:18:53 well, somewhat... basically it would need to ask you things or gather them from dns/etc... 18:19:32 but it might be nice to touch all the places that need it... like inventory/host_vars/playbook/master playbook/dns/2fa-key/openvpn key, etc 18:20:06 anyhow... 18:20:20 any other discussion items? 18:21:49 ok. I signed up to talk about how we deploy new vm's today with ansible... before I start in though, would anyone like to sign up to teach something next week? 18:23:08 I can teach about our RFR process and why it's useful? 18:23:34 sounds good puiterwijk 18:23:49 puiterwijk, RFR? 18:24:00 Request For Resources 18:24:01 puiterwijk: sure, sounds good. we might have some discussion on the stuff on the list adjusting things too... 18:24:10 ah, thanks 18:24:17 nirik: sure. But maybe I can convince people that it's good as it is! :) 18:24:23 could be. :) 18:24:28 Adn otherwise, that will be a good starting point for that conversation 18:24:46 yep. 18:24:55 ok, lets dive into some ansible installing... 18:25:06 #topic Learn about: ansible vm installs in infrastructure - kevin 18:25:17 so we have a pretty automated way to deploy vms using ansible. 18:25:33 If you take a look at most of the playbooks in git, you can see at the top something like: 18:25:57 https://infrastructure.fedoraproject.org/infra/ansible/playbooks/groups/anitya.yml 18:26:04 - include: "/srv/web/infra/ansible/playbooks/include/virt-create.yml myhosts=anitya" 18:26:23 so, that includes that task and calls it with that myhosts variable set to a ansible group. 18:26:33 You can pass host/group/anything ansible knows from it's inventory. 18:26:50 https://infrastructure.fedoraproject.org/infra/ansible/tasks/virt_instance_create.yml 18:27:00 is the core task for installing a new instance. 18:27:20 When an instance is up/already exists, nothing in this task does anything. It's all skipped. 18:27:50 you can see the first thing it does is go to the vmhost(s) defined for myhosts and checks to see if the vm is running/defined. 18:28:10 and then everything else has a when: to make it not run unless the vm is not defined. 18:28:46 So, if it needs to do an install, it makes a lvm volume with the size specified in the variables for the host 18:28:58 then runs the virt-install-command and waits for it to finish. 18:29:27 The virt-install command is defined in inventory/group_vars/all (and can be redefined in hosts variables or whatever if there's a special case) 18:30:27 once it completes installing, it starts it and sets it to autostart by default, waits for ssh to come up and then clears any old ssh host keys that might be laying around 18:31:03 it's all pretty nifty and variable driven. All our vm's are installed via this setup, so we know they are all standard and take the same variables and so forth. 18:31:35 Anyone have questions or comments? 18:32:12 it's cool :] 18:32:23 are there any nice-to-have improvements/changes to the process you would like to try in the future? 18:32:24 I'm going to try and make a screencast of a install so we can show everyone what things look like. 18:32:34 but haven't had time 18:32:43 nirik, that would be most cool 18:33:00 nirik, "clears any old ssh host keys that might be laying around", on batcave? 18:33:06 aikidouke: yeah, there is one I was thinking of... 18:33:25 doteast: yes. since it needs to accept the new ssh host key to connect to it after install. 18:33:45 One cool thing I was thinking of now that we are using qemu-kvm-rhev and can do live migrations again... 18:34:01 sure...just making sure I'm following correctly 18:34:03 thanks 18:34:14 was to modify this to check for the guest on any virthost, if it's not on the one defined in ansible, live migrate it for you to that one. 18:34:32 ie, if you want to migrate something you just change it's virthost in ansible git and run the playbook and it would migrate it. 18:34:36 nirik: how virt_instance_create differs from cloud_setup_basic? 18:34:42 might be tricky to get right, but I think it would be cool. ;) 18:34:48 is the installation done using kickstart or some sort of gold template ? 18:35:24 neat 18:35:33 mirek-hm: cloud_setup_basic does stuff on a new cloud instance thats already up, this actually does a fresh install... not using an image 18:35:39 * pcreech dug around those playbooks one day, very nice work 18:35:39 usernkey: kickstart. 18:35:53 each host has a ks_url with the kickstart to use 18:36:27 ok thx 18:37:07 did I get all the questions? :) anything else? 18:37:42 alrighty then... 18:37:45 #topic Open Floor 18:37:51 anyone have items for open floor? 18:37:56 questions, comments, ideas? 18:38:57 if nothing in a minute, will give you all back 15min of your day. ;) 18:39:28 maybe one meeting soonish, go over goals or plans for 2016? 18:40:00 aikidouke: we usually do those around march (since thats the start of the RH fiscal year), but yeah, never too soon to start... 18:40:26 that sounds soonish to me :) 18:40:33 getting there yeah. 18:41:06 I know I want to revamp monitoring, move a bunch of stuff to pagure, but not sure what else yet 18:42:33 ok, thanks for coming everyone. 18:42:36 #endmeeting