18:00:08 <nirik> #startmeeting Infrastructure (2015-10-15)
18:00:08 <zodbot> Meeting started Thu Oct 15 18:00:08 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:08 <nirik> #meetingname infrastructure
18:00:08 <nirik> #topic aloha
18:00:08 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:08 <zodbot> The meeting name has been set to 'infrastructure'
18:00:08 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:08 <nirik> #topic New folks introductions / Apprentice feedback
18:00:14 <nirik> Greetings everyone.
18:00:15 * threebean is here!
18:00:33 <puffi> Hey everyone.
18:00:35 <nirik> Any new folks like to give a short one line introduction of themselves? Or apprentices with questions or comments?
18:00:39 <aikidouke> howdy
18:00:48 <puiterwijk> shello
18:01:01 * relrod here
18:01:11 * kushal is here
18:02:01 <puffi> An aprentice questio. Is there a full list of all read only machines aprenticies can access?
18:02:06 <smooge> aloha
18:02:15 <dgilmore> hola
18:02:34 <nirik> puffi: well, yeah, in ansible...
18:03:02 <pcreech|work> hola
18:03:41 <nirik> puffi: go to a ansible checkout in batcave01 and do:
18:03:43 <nirik> scripts/hosts_with_var_set -o fas_client_groups=fi-apprentice
18:03:44 <s14ck> hi there o/
18:03:46 <nydel07> here... just a fly on the wall
18:03:48 <lnxslck> hello guys
18:04:09 <puffi> nirik: excellent, Thanks
18:04:32 <nirik> puffi: and... perhaps you (or anyone interested) could add that info to the apprentice pages? :)
18:05:20 * aikidouke thinks that's a great idea...I will do it if puffi doesn't have time or inclination
18:05:36 <nirik> ok, any other new folks or questions? or shall we move on to the gigantic status/info dump we have this week? ;)
18:05:39 <s14ck> puffi, nirik thanks! I was going to say that
18:05:59 <nirik> :)
18:06:09 <nirik> ok, everyone hold on to your irc clients...
18:06:13 <nirik> #topic announcements and information
18:06:13 <nirik> #info Fedora 23 Final Freeze now in effect - Kevin - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/O57Q3CE4Z4M7TA5FFQYASKX6CJGSRYUL/
18:06:13 <nirik> #info [release] pagure: 0.1.31 - pingou
18:06:13 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/XKCH44XS2RH6QFYBTBJMZU3U7XAFKI5K/
18:06:13 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/45ANBM3MABE7IVD7T4MOSGZFDF3Z2L3T/
18:06:16 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/VNM52BXRN2CMQDHC3IIN7OYGSZ3QD7PH/
18:06:19 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/DULZOZ2XGMWYNMFT75V5IDRSOHDCWZ6W/
18:06:22 <nirik> #info [release] pkgdb2: 1.31 - pingou
18:06:24 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/YHLZKAQHZSVAZEP35JMWHOLLDJMRKPJJ/
18:06:29 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/CBPQDH3UXTETIMNA5CUPMT3I52KYWBMP/
18:06:32 <nirik> #info [release] fedmsg_meta_fedora_infrastructure: 0.15.4 - Ralph
18:06:34 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/72ZX2T7GVBHUMTEGYSXFEGLFMU7DC4DR/
18:06:36 * roshi is here
18:06:37 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6ATE2NKCVOGHN3GMH5TDGGNUNPAAHKA6/
18:06:40 <nirik> #info [release] supybot-fedora-0.3.5 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/XZGIPXAC2P4EOQ4JC6GTYMLBPXHT6KKA/
18:06:43 <nirik> #info Upgraded taiga.cloud.fp.o from 1.6.0 to 1.8.0 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/5VJ6RC7JBYMLTELQIMMIJSOFNVLK2TNC/
18:06:46 <nirik> #info [release] python-fedora-0.6.0 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/HRFG4C6W5EQ3ZMGHNJ7RQXODPGKA32DM/
18:06:49 <nirik> #info [release] the-new-hotness: 0.6.3 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/JFBUQJO5D7PISYJJ5OMA2QKGF7ENDQGH/
18:06:52 <nirik> #info [release] bodhi-2.1.2 - Luke - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/GG52IU2WENKP7OBSSMEC5QN5YDUNEGVX/
18:06:57 <nirik> #info [release] datagrepper: 0.7.1 - Ralph
18:06:59 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/PFQK5J7FYZTER5TCCWYJKXZI7ZRVRYOC/
18:07:02 <nirik> #link http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/ZQGHRAWVSRC6LUAP7P5KR2ZDRJYVP5MP/
18:07:05 <nirik> #info [release] fedmsg: 0.16.2 - Ralph - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/6Y7WHRXCNZPVDHRXSUUSKILMLFA4CGSE/
18:07:08 <nirik> #info Firefox addon for askbot sites - Ankur - http://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/Q3UH6H3RBFWCIHCLSYNJEUUMCRSGCGYT/
18:07:11 <nirik> #info kevin out next week (wed->sunday) Can someone run the meeting next week? - kevin
18:07:13 <nirik> #info new gobby/infinote server in place - kevin
18:07:15 <nirik> #info pkgdb2 scm processing is live and announced now - kevin/pingou
18:07:17 <nirik> #info Bunch of tweaking on proxies to get them stable (ongoing) - kevin/patrick/smooge
18:07:19 <nirik> #info We are now down to under 100 tickets in the infrastructure trac! - everyone
18:07:21 <nirik> anything in there folks would like to talk about or add to?
18:07:23 <nirik> Have folks been able to get into the new gobby/infinote server? any feedback? ;)
18:07:26 <puiterwijk> #info Upgraded Ipsilon to 1.1.0 - Patrick
18:07:43 <nirik> ah yeah, and it's been much faster too. ;)
18:07:45 <kushal> nirik, my talk on autocloud?
18:07:48 <aikidouke> I used it today, but just did a quick edit..
18:07:53 <smooge> I am hoping to work on documenting all the changes in proxies once we have stabilized stuff
18:07:54 <nirik> kushal: its toward the end.
18:07:55 <puiterwijk> Oh, yeah,
18:07:56 <aikidouke> seemed smooth
18:08:03 <smooge> as I missed half of what was done while sick
18:08:10 <puiterwijk> #info login issues fixed - Patrick
18:08:13 <puiterwijk> #link http://patrick.uiterwijk.org/2015/10/09/fedora-openid-issues-resolved/
18:08:13 <kushal> nirik, oh, missed the info part
18:08:16 <nirik> just doing info/status then discussion, then learning. ;)
18:08:28 <kushal> nirik, yup, sorry, my mistake.
18:08:32 <nirik> no worries. ;)
18:08:49 <nirik> Oh, I have also info from the pcp folks on how to set it up to monitor apache for us.
18:09:12 <nirik> I was hoping to work on that this morning, but instead I had a net outage and have been dealing with one of our sponsors that switched machines on us. ;)
18:09:34 <nirik> anyhow... on to discussion items:
18:09:41 <nirik> #topic Proposal - Infra Apprentice Workday - aikidouke
18:09:50 <nirik> aikidouke: you walk to talk about this some?
18:09:56 <aikidouke> sure...happy to
18:10:23 <aikidouke> so I am proposing a day when apprentices can collaborate on easy fix tickets
18:10:41 <aikidouke> and updating ansible playbooks with appropriate CSI vars or anything else
18:10:53 <aikidouke> that would be appropriate for an apprentice to do
18:11:03 <aikidouke> timing would be after F23 release
18:11:32 <nirik> sounds good to me. I think we should just pick a day after release sometime for it...
18:11:32 <aikidouke> I'm thinking we could work together in IRC or maybe set up some screen share/casting if that is appropriate
18:11:56 <nirik> yep. Seems reasonable. ;)
18:11:59 <aikidouke> are weekdays or weekends generally better for having master around?
18:12:15 <aikidouke> master(s)
18:12:34 <nirik> probibly want to try and do it during the week I think...
18:12:41 <nirik> perhaps mid november?
18:12:52 <s14ck> week days too
18:13:03 <aikidouke> that was my thought as well. hopefully we can hit a lull between F23 and american turkey day
18:13:13 <nirik> yeah. seems like a good time
18:13:29 <aikidouke> I will send out a note to the DL and put together some items and references
18:13:59 <aikidouke> can anyone think of anything else other than easyfix/csi_vars to work on?
18:14:02 <nirik> sounds good. Should we tenatively say nov 18th or something?
18:14:19 <nirik> I'd be happy to try and triage / make some more easyfixes if we can
18:14:36 <aikidouke> that would be great...(date and triage)
18:14:56 <aikidouke> anyone else have any feedback?
18:14:58 <nirik> perhaps it would also make sense to go over the docs...
18:15:07 <aikidouke> good idea
18:15:24 <lnxslck> aikidouke, maybe before that, first give some info (links, resources) on ansible? for those who haven't played with it?
18:15:29 <purezen> Hi guys, I am keen to get involved in the community.. was looking forward to this meeting.. Glad to be here :)
18:15:39 <lnxslck> hi purezen
18:15:44 <aikidouke> lnxslck: also good idea
18:15:48 <lnxslck> ;)
18:15:55 <nirik> welcome purezen. :)
18:16:30 <purezen> lnxslck: nirik hi everyone :)
18:16:52 <nirik> purezen: are you more interested in sysadmin type tasks or application development? or both?
18:17:32 * pcreech|work thinks it's a good idea (the apprentice day)
18:17:44 <aikidouke> :)
18:18:01 <nirik> hopefully many people can make it...
18:18:11 <nirik> ok, anything else on this?
18:18:16 * s14ck +1 'The Apprentice day'
18:18:18 <aikidouke> not for me...thanks
18:18:50 <nirik> great idea and hopefully it will go well. ;) lets keep up the planning.
18:18:54 <puffi> aikidouke: Sounds like an excellent idea . Thanks for that
18:18:55 <purezen> nirik: So, I have joined an organization lately (Shippable Inc) which uses Docker heavily.. was looking for related work
18:19:26 <nirik> purezen: cool. We are currently not really using docker much... but we do have lots of other stuff. ;)
18:20:06 <nirik> #topic Learn about: autocloud - kushal
18:20:14 <nirik> kushal: you ready to talk about autocloud?
18:20:19 <purezen> nirik: hmm.. nice.. am mostly interested in sysadmin
18:20:27 <nirik> purezen: see us after the meeting in #fedora-admin and we can get you setup. ;)
18:20:39 <purezen> nirik: awesome!
18:20:53 <kushal> nirik, Yes
18:21:21 <kushal> So one of the proposed change for F23 was Two week atomic.
18:21:22 <kushal> https://fedoraproject.org/wiki/Changes/Two_Week_Atomic
18:21:42 <kushal> The idea is to release atomic images in every two week.
18:22:00 <s14ck> wow
18:22:06 <kushal> If you look at the design at https://fedoraproject.org/wiki/Changes/Two_Week_Atomic#Schematic
18:22:44 <kushal> You will find that there is a pink box at the left, which is basically koji + other tools
18:22:53 <kushal> and then a yellow box which does the testing
18:23:06 <kushal> Autocloud came up as a yellow box in this case.
18:23:26 <kushal> if you look at http://autocloud.readthedocs.org/en/latest/setup.html#setup-instruction-on-fedora
18:23:43 <kushal> We have koji running on Fedora infra, and it emits fedmsg(s).
18:23:57 <nirik> readthedocs++ :)
18:24:24 <kushal> Autocloud works based on those fedmsg(s), and it reports back results on fedmsg itself.
18:24:44 <kushal> So, we have a autocloud-common package, which needs to be install in everybox.
18:24:56 <kushal> that is two backends, and two frontend
18:25:32 <kushal> In atomic (and also for fedora cloud) we have both vagrant libvirt and vagrant virtualbox images.
18:26:14 <kushal> It is a known fact and libvirt and virtualbox can not co exist on the same system, and Virtualbox can not also run inside a VM (no nested virt).
18:26:29 <kushal> That is why we have two different backend system.
18:26:55 <kushal> In both the systems, one has to install autocloud-backend package
18:27:25 <kushal> autocloud inside uses a project called Tunir to do the actual testing.
18:27:31 <kushal> http://tunir.readthedocs.org/en/latest/
18:27:38 <kushal> It uses redis.
18:27:48 <kushal> So the next step is to start redis service.
18:28:08 <kushal> and then create a queue of available ports in redis, by running python /usr/share/tunir/createports.py
18:28:54 <kushal> Now we also enable a kill_vagrant command in cron in the libvirt based system, as vagrant-libvirt does not play nicely with the broken vagrant images and timeout option.
18:29:37 <kushal> Now in the libvirt backend we also install vagrant-libvirt package.
18:29:54 <kushal> as we want to run/test the vagrant-libvirt images on this box.
18:30:38 <nirik> so autocloud only tests the vagrant versions? or does it/can it also test the base cloud/atomic?
18:31:11 <kushal> nirik, It also tests the base images, but we don't have setup anything special for it.
18:31:37 <nirik> ah ok.
18:31:43 <kushal> The steps required for Virtualbox is documented at http://tunir.readthedocs.org/en/latest/vagrant.html#how-to-install-virtualbox-and-vagrant
18:32:12 <kushal> The most important point is that for every kernel update, we will have to manually run /etc/init.d/vboxdrv setup after booting in.
18:32:23 <kushal> ^^ only for the Virtualbox to work.
18:33:01 <kushal> We also have a file /etc/autocloud/fedora.txt which contains the actual steps for testing.
18:33:19 <kushal> and a etc/autocloud/autocloud.cfg for the db details etc.
18:33:46 <kushal> To mark the box is only for virtualbox, in the configuration file you write virtualbox = true
18:34:14 <kushal> This are the major steps for setting up, and then we start the servicess
18:34:29 <kushal> In both backends, start fedmsg-hub
18:34:41 <kushal> this will listen for any new build, and put then in a queue.
18:34:54 <kushal> In the backends, you will also want to do systemctl start autocloud
18:35:32 <kushal> on the virtualbox one, it will only test vagrant virtualbox files, all other images including cloud base and atomic qcow2 images will be tested on the libvirt box.
18:35:44 <kushal> The result will be send out to the fedmsg.
18:36:02 <kushal> That means we have usual special endpoints.py file in ansible.
18:36:18 <kushal> Just like any other application in Fedora infra which deals with fedmsg.
18:36:36 <kushal> Now in the two frontends, just start httpd, systemctl start httpd
18:36:55 <kushal> The web app will connect to db for the details. it is written in Flask.
18:37:27 <kushal> #link https://apps.fedoraproject.org/autocloud/jobs/
18:37:38 <kushal> https://apps.fedoraproject.org/autocloud/jobs/ is the running system on the infra.
18:38:00 <kushal> Task ID(s) on the left are the koji task ids (or call it build id).
18:38:13 <kushal> You can see the status, and also the output text.
18:38:41 <kushal> For F23, it already managed to catch the failed network service, and we fixed it.
18:39:04 <kushal> today I saw that in rawhide some other service is broken for cloud base image :)
18:39:06 <nirik> I wonder if we could put (f23) or (rawhide) next to them for more easy viewing... (I know you can go to the task....)
18:39:23 <kushal> nirik, Yes, we will do that part in coming days.
18:39:33 <kushal> nirik, otherwise even we have to keep clicking :(
18:39:33 <nirik> or perhaps just the image name?
18:40:17 <kushal> nirik, Yup, we will do that.
18:40:45 <nirik> cool. Any other plans on the roadmap? This is pretty neat considering how fast you implemented it. ;)
18:40:49 <kushal> nirik, we have bugs open for the same https://github.com/kushaldas/autocloud/issues
18:41:18 <kushal> nirik, yup, the first idea is to have non-gating tests.
18:41:49 <kushal> Means those tests might fail, or pass, but the whole job will depend on other tests.
18:41:58 <nirik> sounds good.
18:42:18 <nirik> Oh if you like we can add this project to easyfix and it can show easyfix tickets for any interested folks too.
18:42:22 <kushal> nirik, and that should be a very easy place to start contributing.
18:42:32 <kushal> nirik, Yup, we should do that.
18:42:50 <nirik> kushal++ :) great job on autocloud
18:42:50 <zodbot> nirik: Karma for kushal changed to 10 (for the f22 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:42:54 <kushal> Btw, the tests are standard Python3 based unittests.
18:43:08 <kushal> So they are generic, can be used anywhere.
18:43:24 <nirik> cool.
18:43:35 <kushal> nirik, and then second super big target (in long time) is to test AMI(s).
18:43:45 * nirik nods.
18:44:22 <kushal> nirik, we are adding up things as required from rel-eng tools, as the release will depend on these results.
18:44:33 <kushal> So still very much ongoing thing.
18:44:36 <nirik> yep.
18:44:40 <kushal> Any more questions?
18:45:00 <nirik> thanks much kushal!
18:45:08 <kushal> nirik, thanks for the stage :)
18:45:31 <nirik> Would someone like to volenteer to teach about something next week? (note: I will not be here next week, so also need someone to run the meeting :)
18:45:36 <s14ck> kushal, when I'll read the docs, then make the questions
18:45:49 <kushal> s14ck, you are most welcome :)
18:46:07 <s14ck> kushal, sounds really interesting
18:46:40 <nirik> #topic Open Floor
18:46:45 <nirik> anyone have anything for open floor?
18:46:52 <nirik> or shall we call it a meeting? :)
18:47:40 <puiterwijk> I had something.. and I lost it
18:47:48 <nirik> ok. I'll try and line up stuff for next week when I am gone...
18:47:51 <nirik> ha
18:48:01 * nirik will close out in a minute if nothing else.
18:48:37 <nirik> ok, thanks for coming everyone!
18:48:43 <nirik> #endmeeting