18:00:03 <nirik> #startmeeting Infrastructure (2015-01-29)
18:00:03 <zodbot> Meeting started Thu Jan 29 18:00:03 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:03 <nirik> #meetingname infrastructure
18:00:03 <nirik> #topic aloha
18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:03 <zodbot> The meeting name has been set to 'infrastructure'
18:00:03 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:10 * pingou here
18:00:14 * sborza here
18:00:20 * fale here
18:00:25 * lmacken 
18:00:34 * puiterwijk here
18:00:37 * michel_slm here
18:00:38 <dgilmore> hola
18:00:41 <taedori> here
18:01:05 <Corey84> .hello corey84
18:01:09 <zodbot> Corey84: corey84 'Corey84' <sheldon.corey@gmail.com>
18:01:11 <Corey84> may be afk at times
18:01:13 <janeznemanic> hi
18:02:36 <nirik> #topic New folks introductions and Apprentice tasks
18:02:46 <nirik> any new folks like to introduce themselves?
18:02:52 <nirik> or apprentices with questions or comments?
18:03:07 * threebean waves
18:03:19 <Corey84> o/
18:03:19 <smooge> here
18:03:37 <puiterwijk> zimbr
18:03:42 <Corey84> no need to "raise hands" here new comers
18:04:19 <nirik> :)
18:04:32 <puiterwijk> errr, sorry for the spam :)
18:05:11 <taedori> Nice meeting you guys. I am from C# .Net world. actually transitioning to linux world. I am learning python. and want to contribute in near future.
18:05:23 <nirik> taedori: welcome. :)
18:05:44 <nirik> taedori: do hang out in our #fedora-apps channel and ask questions... and folks there can point you at some places to start.
18:06:17 <taedori> yes sure.
18:07:34 <nirik> #topic Applications status / discussion
18:07:41 <nirik> any applications news to talk about this week?
18:08:32 <threebean> lots of stuff happened today.. ;p
18:08:33 <nirik> There were some minor releases of a few things...
18:08:38 <nirik> and yeah, today has been busy. ;)
18:08:58 <pingou> #info new anitya just hit prod, 0.1.16
18:09:04 <threebean> we got a new fedmsg message type from koji earlier in the week -- one for when rpms are gpg signed by our signing server, sigul.
18:09:11 <threebean> and an FMN release went out with a new rule to ignore that message..
18:09:26 * mirek-hm is here
18:09:50 <lmacken> threebean: will people need to reset their filters to the defaults to pick it up?
18:09:50 <pingou> #info new koji release sending fedmsg message when a package is signed in sigul (our signing server)
18:10:00 <threebean> lmacken: yup.  or add the new rule by hand.
18:10:06 <pingou> #info new fedmsg_meta release to handle this new message
18:10:25 <pingou> #info new FMN release to be able to include/exclude these new messages
18:11:06 <pingou> #info new python-fedmsg-genacls release to be used on RHEL7 with gitolite3
18:11:27 <Corey84> taedori,  may have some  great pyhton epub  copies for you later
18:11:38 <threebean> #info new the-new-hotness release in staging that auto-adds packages to release-monitoring.org when they're added to Fedora
18:11:59 <mirek-hm> copr sign new packages by GPG key, old packages are still unsigned so we did not put gpgcheck=1 to .repo file.
18:12:00 <nirik> Oh, I also created a mirrormanager2 mirrorlist server... it still needs some work, but we have something to try and test out with now.
18:12:14 <nirik> #info copr now signs new package builds.
18:12:18 <nirik> mirek-hm: great stuff!
18:12:33 <taedori> Thanks Corey84
18:12:37 <pingou> #info new mirrormanager2 based mirrorlist server
18:13:08 <Corey84> nirik,  if my server is not showing in manager2  list  let me know  I plan to go  full /alt/pub  when Southern_Gentlem  drops the initial 21 updates next week
18:13:15 <nirik> threebean: is that the thing that sometimes says "new upstream release of foo, but cannot figure out what it's called in Fedora?"
18:13:33 <Corey84> im presently showing private only i believe
18:13:46 <nirik> Corey84: this is just the mirrorlist part... the part that sends users the list of mirrors. ;)
18:13:56 <nirik> but yeah, we can check yours
18:14:09 <threebean> nirik: no, that's the other was around.. the project exists in release-monitoring.org, but there's no mapping to the project's name in Fedora.
18:14:34 <threebean> we could/should write a script that tries to get that mapping as close to 1-to-1 as possible.
18:14:36 <nirik> threebean: ok. I noticed that I think on a 'chrome' package... which of course we don't ship
18:14:43 <threebean> right
18:14:46 <Corey84> kk
18:14:57 <lmacken> the recent koji update also included the runroot plugin, which once configured will let us do things like mash & compose atomic trees
18:15:07 <Corey84> was thinking you meant mirrormanager-client v2  my bad
18:15:12 <nirik> lmacken: yep. Yea!
18:15:23 <pingou> lmacken: cool!
18:15:30 <nirik> Corey84: the rest is coming too, we just are doing the lists first. ;)
18:15:37 * tflink is getting around to adding changes to the ansible playbooks to support local taskotron deployments
18:15:49 <tflink> not sure if that's of much interest to the folks here, though
18:16:06 <Corey84> my bad  ill get back in the carriage and stop tugging the on the bridle :P
18:16:10 <nirik> tflink: oh, I noticed the other day that autoqa was still/again up... is that supposed to still be around/
18:16:12 <nirik> ?
18:16:16 <nirik> autoqa01 that si
18:16:24 <Corey84> tflink,  need a hand let me know
18:16:32 <tflink> no, I don't think we still need it
18:16:47 <tflink> making that more definite - no, we don't still need it
18:16:58 <nirik> cool. I can quash it, or you can
18:17:35 <nirik> ok, anything else application wise?
18:18:00 <nirik> It seems out hyperkitty is delayed, but hopefully will be possible by pycon
18:18:07 <nirik> There was a blog post from abompard_
18:18:34 <Corey84> pycon is when again
18:18:35 <nirik> Not sure where we are on bodhi2... need packaging work?
18:18:40 <tflink> nirik: I'm not sure what the process for that is off the top of my head, if you have the spare cycles, feel free. otherwise, I'll figure it out
18:18:45 <lmacken> Corey84: apr 9-17
18:19:10 <threebean> http://aurelien.bompard.org/post/2015/01/23/News-of-the-Mailman3-front
18:19:15 <nirik> mirek-hm: new cloud is looking promising this time? ;)
18:19:41 <lmacken> nirik: we've been working on some of the email plumbing this week, and I need to forward-port a handful of features that crept into bodhi1 before we get it in staging
18:19:50 <mirek-hm> nirik: no promises. I'm again fighting with rabbitmq
18:19:59 <nirik> lmacken: cool. does the runroot stuff help you out any?
18:20:18 <nirik> mirek-hm: ok. ;( at least we seem to be getting further and iterating faster. ;)
18:20:34 <mirek-hm> nirik: but OS have my FULL attention and top priority for me
18:20:47 <lmacken> nirik: yeah, once it's configured I'll be able to use it to mash pretty much immediately. Assuming in the config file we have it nfs mount /mnt/koji/mash/
18:20:52 <nirik> yeah, I really really want to decommission the old one
18:21:11 <puiterwijk> mirek-hm: if you need any help, just let me know. I'm more than happy to help wherever I can
18:21:22 <nirik> mirek-hm: me too.
18:21:30 <tflink> is the hoped-for deployment date of bodhi2 still during the f22 cycle?
18:21:52 <lmacken> tflink: That's what we're shooting for
18:22:06 <nirik> tflink: I think we should know more once we have a actual stg instance...
18:22:08 * tflink questions the wisdom of doing that mid-release
18:22:17 <nirik> then we can see what needs to be done and how much issue we are hitting
18:22:42 <nirik> yeah, we want to make sure it's ready.
18:22:55 <tflink> but I've not been involved with bodhi2, so I might just be overly-cautious
18:23:37 <mirek-hm> puiterwijk: nirik: I will ask when I reach dead end. But it is most time consuming. Change something run packstack, wait 20 minutes; see if the change changed something and if changed something then check what else was broken this time... and during that 20 minute you should be strong enough to not look on G+ or FB or you will not return to work after those 20 minutes :)
18:23:51 <nirik> threebean: whats the status/timeline on FMN by default + anytia? still shaking out issues?
18:24:03 <lmacken> as long as the bodhi API remains stable, we'll have to see how many other things explode to really see how easily we can pull the switch
18:24:17 <pingou> nirik: I still have 1 issue to fix on anitya for the backend
18:24:25 <nirik> mirek-hm: yeah :)
18:24:33 <threebean> nirik: moving as quick as we can before freeze.
18:24:45 <Corey84> +1 on tflink reluctance
18:24:46 <threebean> fmn is already 'default' in so far as new packagers are automatically set up.
18:24:50 <nirik> cool cool. Just wanted to know if we needed to punt further.
18:25:15 <threebean> I'm just looking for a round tuit to edit and run the script to add all the existing packagers (to fmn).
18:25:22 <tflink> I'm just worried about the possibility of something going wrong during freeze and causing slips
18:25:30 <Corey84> not that we can't do it or do it gracefully but why needlessly add the stress
18:25:34 <nirik> tflink: a completely valid concern
18:25:53 <Corey84> more worried of slips personally not the freeze
18:25:56 <threebean> pingou: remind me what the last issue on anitya is?
18:26:06 <pingou> threebean: urls with * in them
18:26:14 * threebean nods
18:26:21 <threebean> and we can't take hotness to prod without that?
18:26:35 <pingou> we could but I'd prefer to try to tackle it before
18:26:41 <threebean> sure thing.
18:26:46 <pingou> but it's been running like this for a little time now :)
18:26:59 <nirik> alright, any other application side things? or shall we move along to sysadmin?
18:27:29 <nirik> #topic Sysadmin status / discussion
18:27:40 <nirik> So lots of fun this week...
18:27:59 <nirik> #info short mirrorlist outage on tuesday. We think cause was a bad pkl, but not sure.
18:28:15 <nirik> #info glibc updated for ghost and all servers rebooted on tuesday.
18:28:46 <nirik> #info pingou and bochecha have been working on pkgs01.stg and it's ready for wide testing now! Hurray!
18:29:23 <michel_slm> hurray!
18:29:27 <nirik> #info fedoramagazine had some dns issues, but they should be all cleared up
18:29:47 <nirik> #info proxy03 and proxy04 were reinstalled rhel7/ansible and so far seem to be working well.
18:30:10 <nirik> #info koji01/02 were installed as rhel7/ansible and we switched to them in production.
18:30:28 <nirik> #info all builders and hubs were updated to latest git head koji packages
18:30:42 <Corey84> was that the one we were all checking dnssec tues for nirik ?
18:30:44 <nirik> so, we should have that runroot plugin (but it's not configured yet)
18:31:01 <nirik> Corey84: oh yeah...
18:31:16 <nirik> #info letoto has loaded all fedora folks gpg keys into dns now.
18:31:28 <nirik> ^ thats the dnssec thing.
18:32:12 <nirik> #info down to just 44 hosts in puppet.
18:32:36 <nirik> #info relrod fixed up ntp on hosts
18:32:56 <nirik> we are going to see if the old 'guest can get its time from the host' thing works or if we need to run ntp on guests too
18:33:25 <nirik> I fixed up some idempotentcy issues with ansible playbooks last weeked.
18:33:30 <nirik> there's still more to clean up though.
18:33:53 <threebean> any outstanding projects left for the port to ansible?
18:34:11 <michel_slm> is there a list of Ansible tasks somewhere?
18:34:12 <threebean> we can slowly move the proxies over as we gain confidence that they're not broken.
18:34:21 <nirik> mostly I think we are down to remote sites where reinstalling is a pain
18:34:47 <nirik> oh, I made a db01.stg a while back, but still there's dbs on db02.stg that need migrated.
18:35:10 <nirik> Hosted is still on the agenda. That will be some work.
18:35:26 <nirik> we need to build out trac for rhel7 for it, and then setup playbooks, etc.
18:35:44 <nirik> we need a kojipkgs playbook/setup.
18:35:58 <nirik> (thats just apache/squid converted from puppet, shouldn't be too hard)
18:36:10 <nirik> people still needs doing
18:36:11 <smooge> threebean, I started on people and got stuck.. I think the proxy stuff might help me
18:36:20 <threebean> nice, nice.  can test kojipkgs in stg too.
18:36:20 <nirik> torrent
18:36:38 <smooge> #info DNS zones have been cleaned up so they all refer to the same set of proxies.
18:36:47 <nirik> and some releng machines... which need to stay rhel6 as bodhi1 needs to run on them for now.
18:37:07 <smooge> #info DNS templates code has been fixed so that zones don't give "this serial is out of bounds rejecting zone."
18:37:12 <nirik> we are reaching the end. ;)
18:37:37 <threebean> :p
18:37:40 <threebean> FAD made a big difference.
18:37:45 <dgilmore> nirik: torrent needs to go away or we need to redo it
18:37:57 <nirik> http://paste.fedoraproject.org/177673/42255665 is all the hosts left in puppet.
18:38:09 <nirik> dgilmore: I wish we could drop it, but everytime we suggest it, tons of people yell.
18:38:39 <dgilmore> nirik: right. we did talk of using a service provided by ibiblio at some point
18:38:55 <nirik> yeah, and then the guy who was doing that left ibiblio and the service died. ;(
18:39:10 <dgilmore> :( good thing we did not move then
18:39:16 <nirik> yeah, no kidding
18:39:27 <dgilmore> so I guess torrent needs someone to redesign it
18:39:28 <nirik> anyhow, we can investigate. perhaps there's better software now
18:39:31 <nirik> yep
18:39:52 <dgilmore> I have a few complaints about the usage
18:40:00 <nirik> #info torrent setup needs a redesign.
18:40:18 <nirik> anything else on the sysadmin side?
18:40:37 <dgilmore> nirik: one quick thing
18:40:43 <dgilmore> maybe not meeting worthy
18:40:49 <nirik> sure thing, go ahead
18:41:02 <dgilmore> need to work out a way to get pbrobinson onto the builders
18:41:24 <dgilmore> I see two ways, add him to sysadmin-main or add fas etc on to the builders
18:41:26 <nirik> dgilmore: we could easily add fasclient back and add sysadmin-build or whatever
18:41:30 <nirik> yeah
18:41:41 <dgilmore> and let sysadmin-releng login and sudo
18:42:13 <dgilmore> I do not mind too much how we do it but it has to be done
18:42:39 <dgilmore> i guess another option is have a releng-lockbox
18:42:44 <nirik> I'd be fine adding fas back for now... not that I don't think he would be fine in main, but typically we wait until someone has done some big project/work before adding them.
18:42:49 <dgilmore> and allow root from it to ssh to the builders
18:43:09 <smooge> I thought sysadmin-releng had sudo at one point
18:43:11 <nirik> that seems like it might be more complicated...
18:43:24 <Corey84> dgilmore,  torrents need to go ?
18:43:34 <dgilmore> smooge: it was removed when skvidal redesigned the builder setup
18:43:36 <smooge> yes. I have a long diatribe on them
18:43:45 <smooge> dgilmore, got it
18:43:52 <nirik> yeah, builders currently have just root. Nothing else.
18:44:00 <smooge> Corey84,  yes. I have a long diatribe on them. After meeting
18:44:04 <Corey84> if someone wants to help with the layout side (ui)  I can throw some time in on that
18:44:08 <dgilmore> I kinda like not having accounts on it
18:44:17 <Corey84> smooge, kk
18:44:30 <dgilmore> but setting up a box just to allow some folks access seems a bit of overkill
18:44:32 <Corey84> accounts on torrents ?  new to me
18:44:43 <smooge> no we moved to build system :)
18:44:45 <dgilmore> Corey84: no accounts on the builders
18:44:55 <Corey84> builder servers?
18:45:03 <dgilmore> yes
18:45:04 <nirik> dgilmore: right. I am leaning toward fas on them again, but we could discuss more out of meeting?
18:45:24 <dgilmore> nirik: sure. just wanted to bring up we need to do something
18:45:33 <dgilmore> details can be sorted
18:45:43 <nirik> ok. yeah, I also need to get together with him and work on ansiblizing aarch64 boxes
18:45:49 <nirik> yeap
18:45:52 <nirik> #topic Upcoming Tasks/Items
18:45:52 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:46:01 <nirik> anyone have any upcoming items they want to schedule or discuss?
18:46:15 <nirik> note that smooge and I will be at our main datacenter the week of feb 9th
18:46:23 <nirik> branching is also that week.
18:46:28 <pingou> FOSDEM this week-end
18:46:29 <threebean> all those EU events are coming up soon, right?
18:46:32 <puiterwijk> Staging upgrade to Ipsilon can finally take place now that we have a EPEL7 build, so will be doing that this week or early next week
18:46:34 <pingou> DevConf the week-end after
18:46:41 <Corey84> +1 dgilmore  a bit overkill it seems
18:46:41 <nirik> threebean: yeah
18:46:53 <threebean> pingou, puiterwijk: have fun :)
18:47:01 <pingou> we were discussing if we could move pkgs01.stg to prod on Monday
18:47:02 <puiterwijk> CfgMgmtCamp next monday and tuesday, and infra.next on wednesday
18:47:11 <pingou> giving us 2 days to fix things before travel to DevConf
18:47:13 <nirik> puiterwijk: excellent. was going to ask about that... let me know if there's anything I can do to help with it.
18:47:15 <pingou> threebean: will do :)
18:47:23 <dgilmore> nirik: might try to get with you and make sure the aarch64 boxes are rebuilt then
18:47:39 <puiterwijk> nirik: will do, though most of it is just going to be porting templates and configuration
18:47:45 <Corey84> Cydrobolt,   and I were on the mail list about some upcoming hackathons and makerspace  sponsorships  ( more an ambass. thing but figured i'd mention ntl)
18:47:46 <dgilmore> nirik: pbrobinson and I will be working together in Brno that week
18:47:47 <nirik> dgilmore: yeah, they need to be... one of the things we are going to do in phx2 is move them tho to a common rack
18:48:20 <puiterwijk> (Ipsilon might be "FedOAuth 3", but it's a major change, so none of the config or ui templates look alike)
18:48:27 <nirik> cool.
18:48:33 <dgilmore> need to get a plan to redo all teh secondary arch setup and ansibilise it all
18:48:33 <pingou> puiterwijk: nice :)
18:48:51 <Corey84> +1 ansibilise secondary
18:48:57 <nirik> threebean: you game to talk about github2fedmsg for a few?
18:49:01 <threebean> sure :)
18:49:09 <nirik> #topic learn an app - github2fedmsg
18:49:10 * pingou has a meeting in 10minutes
18:49:19 <nirik> yeah, sorry, not much time...
18:49:27 <Corey84> same here ....use github /bitbucket religously
18:49:27 <threebean> so, github2fedmsg is a web service that bridges github activity onto the fedmsg bus
18:49:45 <threebean> it has a little self-service dashboard which you can access here
18:49:47 <threebean> https://apps.fedoraproject.org/github2fedmsg
18:50:08 <threebean> you login with FAS, then login with GitHub (it keeps track of the association between your accounts there)
18:50:14 <Corey84> so you setup as an appication (like tfa on GH token) ?
18:50:36 <Corey84> s/appication/application
18:50:42 <threebean> Corey84: right.  yeah, and it's already setup with an application key in our infra.  for users, they don't have to worry about that part.
18:50:54 <Corey84> sweet
18:51:10 <threebean> it uses that application key and the oauth token it gets when you login to install webhooks on the github repos you choose.
18:51:14 <Corey84> nice had to walk some kids through that last night not fun
18:51:22 <threebean> it can be a mess ;)
18:51:47 <threebean> so then with those installed webhooks, github will POST back to the webapp in our infrastructure about events as they happen.
18:51:49 <Corey84> whoever set all that up so nicely deserves a brew or two :)
18:52:11 <threebean> the github2fedmsg app receives those POSTs, verifies their signature is really from github, then re-signs them with a fedmsg key and broadcasts them on our bus.
18:52:14 <Corey84> so just sub  to the respective whooks
18:52:16 <Corey84> ?
18:52:25 * threebean nods
18:52:29 <threebean> yup
18:52:42 <nirik> this can be any github repo anyone cares about? or you have to be an admin for that github project to add the hooks?
18:52:55 <threebean> nirik: hm, you have to have admin on that github project, yes.
18:52:58 * Corey84 feels a warp drive speed increase in workflow coming when fully used
18:53:32 <threebean> nirik: so you can add your "own" repos as well as repos of any organization you have sufficient rights in.
18:53:36 <Corey84> much like  sponsorship for maintainers tho someone can add you or whitelist you tho
18:54:09 * Corey84 will be initiating later today when he has 5 mins to spare
18:54:36 <threebean> the source code for it all is here -> https://github.com/fedora-infra/github2fedmsg
18:55:09 <threebean> this is maybe the most interesting block -> https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L69-L136
18:55:16 <threebean> any questions or anything?
18:55:38 <nirik> agpl ? I guess we never should hotfix it, but make new releases. ;)
18:55:52 <threebean> ah, it's agpl with that extra clause iirc.
18:56:03 <threebean> the one that gives us 30 days to get our srpms in line.
18:56:10 <nirik> oh right
18:56:33 <nirik> cool. Thanks threebean!
18:56:37 <threebean> np!
18:56:38 <nirik> #topic Open Floor
18:56:49 <nirik> just a few minutes left for open floor. Anyone have anything?
18:56:55 <tflink> who all is going to be at devconf next week?
18:57:08 <nirik> puiterwijk and pingou I think...
18:57:28 <puiterwijk> stickster as well, I think
18:57:40 <nirik> and dgilmore ?
18:57:45 <puiterwijk> oh, and I heard jwb is there
18:57:49 <puiterwijk> I think Dennis as well yeah
18:58:03 <tflink> k, I want to run some of our taskotron plans past someone to make sure they aren't any crazier than I think they are and that'd be easier in person
18:58:40 <nirik> tflink: BTW, I pinged on that fw ticket. Hopefully we will get something soon. That same ticket also has the ipmi for qa hosts for beaker in it.
18:58:43 <michel_slm> anyone here going to FOSS Asia the month after that?
18:59:01 <puiterwijk> tflink: I'd be glad to, just let me know when you arrive/have time
18:59:10 <tflink> also, the secarch folks are asking for blockerbugs instances. I'll know more about it after devconf but figured that I'd mention it here
18:59:34 <nirik> oh? seperate ones?
18:59:37 * dgilmore will be at fosdem and devconf
19:00:04 <tflink> nirik: at this point, it'd have to be. adding multiple arch support to the codebase isn't going to happen before f22 starts up
19:00:27 <smooge> michel_slm, I don't think so.
19:00:31 <nirik> tflink: ok
19:00:33 <puiterwijk> for those interrested, I'm giving a talk at saturday on centralized authn
19:00:34 <tflink> puiterwijk: I'll be in brno from feb 5-18
19:00:54 <puiterwijk> tflink: I from feb 5 to 9, so we should be able to find a time
19:01:27 <nirik> cool. well, if nothing else will close out the meeting in a minute or so
19:02:22 <nirik> thanks for coming everyone.
19:02:24 <nirik> #endmeeting