18:00:08 #startmeeting Infrastructure (2015-10-15) 18:00:08 Meeting started Thu Oct 15 18:00:08 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:08 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:08 #meetingname infrastructure 18:00:08 #topic aloha 18:00:08 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:08 The meeting name has been set to 'infrastructure' 18:00:08 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:08 #topic New folks introductions / Apprentice feedback 18:00:14 Greetings everyone. 18:00:15 * threebean is here! 18:00:33 Hey everyone. 18:00:35 Any new folks like to give a short one line introduction of themselves? Or apprentices with questions or comments? 18:00:39 howdy 18:00:48 shello 18:01:01 * relrod here 18:01:11 * kushal is here 18:02:01 An aprentice questio. Is there a full list of all read only machines aprenticies can access? 18:02:06 aloha 18:02:15 hola 18:02:34 puffi: well, yeah, in ansible... 18:03:02 hola 18:03:41 puffi: go to a ansible checkout in batcave01 and do: 18:03:43 scripts/hosts_with_var_set -o fas_client_groups=fi-apprentice 18:03:44 hi there o/ 18:03:46 here... just a fly on the wall 18:03:48 hello guys 18:04:09 nirik: excellent, Thanks 18:04:32 puffi: and... perhaps you (or anyone interested) could add that info to the apprentice pages? :) 18:05:20 * aikidouke thinks that's a great idea...I will do it if puffi doesn't have time or inclination 18:05:36 ok, any other new folks or questions? or shall we move on to the gigantic status/info dump we have this week? ;) 18:05:39 puffi, nirik thanks! I was going to say that 18:05:59 :) 18:06:09 ok, everyone hold on to your irc clients... 18:06:13 #topic announcements and information 18:06:13 #info Fedora 23 Final Freeze now in effect - Kevin - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/O57Q3CE4Z4M7TA5FFQYASKX6CJGSRYUL/ 18:06:13 #info [release] pagure: 0.1.31 - pingou 18:06:13 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/XKCH44XS2RH6QFYBTBJMZU3U7XAFKI5K/ 18:06:13 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/45ANBM3MABE7IVD7T4MOSGZFDF3Z2L3T/ 18:06:16 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/VNM52BXRN2CMQDHC3IIN7OYGSZ3QD7PH/ 18:06:19 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/DULZOZ2XGMWYNMFT75V5IDRSOHDCWZ6W/ 18:06:22 #info [release] pkgdb2: 1.31 - pingou 18:06:24 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/YHLZKAQHZSVAZEP35JMWHOLLDJMRKPJJ/ 18:06:29 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/CBPQDH3UXTETIMNA5CUPMT3I52KYWBMP/ 18:06:32 #info [release] fedmsg_meta_fedora_infrastructure: 0.15.4 - Ralph 18:06:34 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/72ZX2T7GVBHUMTEGYSXFEGLFMU7DC4DR/ 18:06:36 * roshi is here 18:06:37 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6ATE2NKCVOGHN3GMH5TDGGNUNPAAHKA6/ 18:06:40 #info [release] supybot-fedora-0.3.5 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/XZGIPXAC2P4EOQ4JC6GTYMLBPXHT6KKA/ 18:06:43 #info Upgraded taiga.cloud.fp.o from 1.6.0 to 1.8.0 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/5VJ6RC7JBYMLTELQIMMIJSOFNVLK2TNC/ 18:06:46 #info [release] python-fedora-0.6.0 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/HRFG4C6W5EQ3ZMGHNJ7RQXODPGKA32DM/ 18:06:49 #info [release] the-new-hotness: 0.6.3 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/JFBUQJO5D7PISYJJ5OMA2QKGF7ENDQGH/ 18:06:52 #info [release] bodhi-2.1.2 - Luke - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/GG52IU2WENKP7OBSSMEC5QN5YDUNEGVX/ 18:06:57 #info [release] datagrepper: 0.7.1 - Ralph 18:06:59 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/PFQK5J7FYZTER5TCCWYJKXZI7ZRVRYOC/ 18:07:02 #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/ZQGHRAWVSRC6LUAP7P5KR2ZDRJYVP5MP/ 18:07:05 #info [release] fedmsg: 0.16.2 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6Y7WHRXCNZPVDHRXSUUSKILMLFA4CGSE/ 18:07:08 #info Firefox addon for askbot sites - Ankur - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/Q3UH6H3RBFWCIHCLSYNJEUUMCRSGCGYT/ 18:07:11 #info kevin out next week (wed->sunday) Can someone run the meeting next week? - kevin 18:07:13 #info new gobby/infinote server in place - kevin 18:07:15 #info pkgdb2 scm processing is live and announced now - kevin/pingou 18:07:17 #info Bunch of tweaking on proxies to get them stable (ongoing) - kevin/patrick/smooge 18:07:19 #info We are now down to under 100 tickets in the infrastructure trac! - everyone 18:07:21 anything in there folks would like to talk about or add to? 18:07:23 Have folks been able to get into the new gobby/infinote server? any feedback? ;) 18:07:26 #info Upgraded Ipsilon to 1.1.0 - Patrick 18:07:43 ah yeah, and it's been much faster too. ;) 18:07:45 nirik, my talk on autocloud? 18:07:48 I used it today, but just did a quick edit.. 18:07:53 I am hoping to work on documenting all the changes in proxies once we have stabilized stuff 18:07:54 kushal: its toward the end. 18:07:55 Oh, yeah, 18:07:56 seemed smooth 18:08:03 as I missed half of what was done while sick 18:08:10 #info login issues fixed - Patrick 18:08:13 #link http://patrick.uiterwijk.org/2015/10/09/fedora-openid-issues-resolved/ 18:08:13 nirik, oh, missed the info part 18:08:16 just doing info/status then discussion, then learning. ;) 18:08:28 nirik, yup, sorry, my mistake. 18:08:32 no worries. ;) 18:08:49 Oh, I have also info from the pcp folks on how to set it up to monitor apache for us. 18:09:12 I was hoping to work on that this morning, but instead I had a net outage and have been dealing with one of our sponsors that switched machines on us. ;) 18:09:34 anyhow... on to discussion items: 18:09:41 #topic Proposal - Infra Apprentice Workday - aikidouke 18:09:50 aikidouke: you walk to talk about this some? 18:09:56 sure...happy to 18:10:23 so I am proposing a day when apprentices can collaborate on easy fix tickets 18:10:41 and updating ansible playbooks with appropriate CSI vars or anything else 18:10:53 that would be appropriate for an apprentice to do 18:11:03 timing would be after F23 release 18:11:32 sounds good to me. I think we should just pick a day after release sometime for it... 18:11:32 I'm thinking we could work together in IRC or maybe set up some screen share/casting if that is appropriate 18:11:56 yep. Seems reasonable. ;) 18:11:59 are weekdays or weekends generally better for having master around? 18:12:15 master(s) 18:12:34 probibly want to try and do it during the week I think... 18:12:41 perhaps mid november? 18:12:52 week days too 18:13:03 that was my thought as well. hopefully we can hit a lull between F23 and american turkey day 18:13:13 yeah. seems like a good time 18:13:29 I will send out a note to the DL and put together some items and references 18:13:59 can anyone think of anything else other than easyfix/csi_vars to work on? 18:14:02 sounds good. Should we tenatively say nov 18th or something? 18:14:19 I'd be happy to try and triage / make some more easyfixes if we can 18:14:36 that would be great...(date and triage) 18:14:56 anyone else have any feedback? 18:14:58 perhaps it would also make sense to go over the docs... 18:15:07 good idea 18:15:24 aikidouke, maybe before that, first give some info (links, resources) on ansible? for those who haven't played with it? 18:15:29 Hi guys, I am keen to get involved in the community.. was looking forward to this meeting.. Glad to be here :) 18:15:39 hi purezen 18:15:44 lnxslck: also good idea 18:15:48 ;) 18:15:55 welcome purezen. :) 18:16:30 lnxslck: nirik hi everyone :) 18:16:52 purezen: are you more interested in sysadmin type tasks or application development? or both? 18:17:32 * pcreech|work thinks it's a good idea (the apprentice day) 18:17:44 :) 18:18:01 hopefully many people can make it... 18:18:11 ok, anything else on this? 18:18:16 * s14ck +1 'The Apprentice day' 18:18:18 not for me...thanks 18:18:50 great idea and hopefully it will go well. ;) lets keep up the planning. 18:18:54 aikidouke: Sounds like an excellent idea . Thanks for that 18:18:55 nirik: So, I have joined an organization lately (Shippable Inc) which uses Docker heavily.. was looking for related work 18:19:26 purezen: cool. We are currently not really using docker much... but we do have lots of other stuff. ;) 18:20:06 #topic Learn about: autocloud - kushal 18:20:14 kushal: you ready to talk about autocloud? 18:20:19 nirik: hmm.. nice.. am mostly interested in sysadmin 18:20:27 purezen: see us after the meeting in #fedora-admin and we can get you setup. ;) 18:20:39 nirik: awesome! 18:20:53 nirik, Yes 18:21:21 So one of the proposed change for F23 was Two week atomic. 18:21:22 https://fedoraproject.org/wiki/Changes/Two_Week_Atomic 18:21:42 The idea is to release atomic images in every two week. 18:22:00 wow 18:22:06 If you look at the design at https://fedoraproject.org/wiki/Changes/Two_Week_Atomic#Schematic 18:22:44 You will find that there is a pink box at the left, which is basically koji + other tools 18:22:53 and then a yellow box which does the testing 18:23:06 Autocloud came up as a yellow box in this case. 18:23:26 if you look at http://autocloud.readthedocs.org/en/latest/setup.html#setup-instruction-on-fedora 18:23:43 We have koji running on Fedora infra, and it emits fedmsg(s). 18:23:57 readthedocs++ :) 18:24:24 Autocloud works based on those fedmsg(s), and it reports back results on fedmsg itself. 18:24:44 So, we have a autocloud-common package, which needs to be install in everybox. 18:24:56 that is two backends, and two frontend 18:25:32 In atomic (and also for fedora cloud) we have both vagrant libvirt and vagrant virtualbox images. 18:26:14 It is a known fact and libvirt and virtualbox can not co exist on the same system, and Virtualbox can not also run inside a VM (no nested virt). 18:26:29 That is why we have two different backend system. 18:26:55 In both the systems, one has to install autocloud-backend package 18:27:25 autocloud inside uses a project called Tunir to do the actual testing. 18:27:31 http://tunir.readthedocs.org/en/latest/ 18:27:38 It uses redis. 18:27:48 So the next step is to start redis service. 18:28:08 and then create a queue of available ports in redis, by running python /usr/share/tunir/createports.py 18:28:54 Now we also enable a kill_vagrant command in cron in the libvirt based system, as vagrant-libvirt does not play nicely with the broken vagrant images and timeout option. 18:29:37 Now in the libvirt backend we also install vagrant-libvirt package. 18:29:54 as we want to run/test the vagrant-libvirt images on this box. 18:30:38 so autocloud only tests the vagrant versions? or does it/can it also test the base cloud/atomic? 18:31:11 nirik, It also tests the base images, but we don't have setup anything special for it. 18:31:37 ah ok. 18:31:43 The steps required for Virtualbox is documented at http://tunir.readthedocs.org/en/latest/vagrant.html#how-to-install-virtualbox-and-vagrant 18:32:12 The most important point is that for every kernel update, we will have to manually run /etc/init.d/vboxdrv setup after booting in. 18:32:23 ^^ only for the Virtualbox to work. 18:33:01 We also have a file /etc/autocloud/fedora.txt which contains the actual steps for testing. 18:33:19 and a etc/autocloud/autocloud.cfg for the db details etc. 18:33:46 To mark the box is only for virtualbox, in the configuration file you write virtualbox = true 18:34:14 This are the major steps for setting up, and then we start the servicess 18:34:29 In both backends, start fedmsg-hub 18:34:41 this will listen for any new build, and put then in a queue. 18:34:54 In the backends, you will also want to do systemctl start autocloud 18:35:32 on the virtualbox one, it will only test vagrant virtualbox files, all other images including cloud base and atomic qcow2 images will be tested on the libvirt box. 18:35:44 The result will be send out to the fedmsg. 18:36:02 That means we have usual special endpoints.py file in ansible. 18:36:18 Just like any other application in Fedora infra which deals with fedmsg. 18:36:36 Now in the two frontends, just start httpd, systemctl start httpd 18:36:55 The web app will connect to db for the details. it is written in Flask. 18:37:27 #link https://apps.fedoraproject.org/autocloud/jobs/ 18:37:38 https://apps.fedoraproject.org/autocloud/jobs/ is the running system on the infra. 18:38:00 Task ID(s) on the left are the koji task ids (or call it build id). 18:38:13 You can see the status, and also the output text. 18:38:41 For F23, it already managed to catch the failed network service, and we fixed it. 18:39:04 today I saw that in rawhide some other service is broken for cloud base image :) 18:39:06 I wonder if we could put (f23) or (rawhide) next to them for more easy viewing... (I know you can go to the task....) 18:39:23 nirik, Yes, we will do that part in coming days. 18:39:33 nirik, otherwise even we have to keep clicking :( 18:39:33 or perhaps just the image name? 18:40:17 nirik, Yup, we will do that. 18:40:45 cool. Any other plans on the roadmap? This is pretty neat considering how fast you implemented it. ;) 18:40:49 nirik, we have bugs open for the same https://github.com/kushaldas/autocloud/issues 18:41:18 nirik, yup, the first idea is to have non-gating tests. 18:41:49 Means those tests might fail, or pass, but the whole job will depend on other tests. 18:41:58 sounds good. 18:42:18 Oh if you like we can add this project to easyfix and it can show easyfix tickets for any interested folks too. 18:42:22 nirik, and that should be a very easy place to start contributing. 18:42:32 nirik, Yup, we should do that. 18:42:50 kushal++ :) great job on autocloud 18:42:50 nirik: Karma for kushal changed to 10 (for the f22 release cycle): https://badges.fedoraproject.org/tags/cookie/any 18:42:54 Btw, the tests are standard Python3 based unittests. 18:43:08 So they are generic, can be used anywhere. 18:43:24 cool. 18:43:35 nirik, and then second super big target (in long time) is to test AMI(s). 18:43:45 * nirik nods. 18:44:22 nirik, we are adding up things as required from rel-eng tools, as the release will depend on these results. 18:44:33 So still very much ongoing thing. 18:44:36 yep. 18:44:40 Any more questions? 18:45:00 thanks much kushal! 18:45:08 nirik, thanks for the stage :) 18:45:31 Would someone like to volenteer to teach about something next week? (note: I will not be here next week, so also need someone to run the meeting :) 18:45:36 kushal, when I'll read the docs, then make the questions 18:45:49 s14ck, you are most welcome :) 18:46:07 kushal, sounds really interesting 18:46:40 #topic Open Floor 18:46:45 anyone have anything for open floor? 18:46:52 or shall we call it a meeting? :) 18:47:40 I had something.. and I lost it 18:47:48 ok. I'll try and line up stuff for next week when I am gone... 18:47:51 ha 18:48:01 * nirik will close out in a minute if nothing else. 18:48:37 ok, thanks for coming everyone! 18:48:43 #endmeeting