18:00:06 <smooge> #startmeeting Infrastructure (2018-01-04)
18:00:06 <zodbot> Meeting started Thu Jan  4 18:00:06 2018 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:06 <zodbot> The meeting name has been set to 'infrastructure_(2018-01-04)'
18:00:06 <smooge> #meetingname infrastructure
18:00:06 <zodbot> The meeting name has been set to 'infrastructure'
18:00:06 <smooge> #topic aloha
18:00:06 <smooge> #chair smooge relrod nirik dgilmore threebean pingou puiterwijk pbrobinson maxamillion
18:00:06 <zodbot> Current chairs: dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:11 <smooge> Good day everyone
18:00:23 <puiterwijk> hello
18:00:25 <pingou> Good Morning Everyone
18:00:27 <nirik> morning
18:00:36 <tflink> hello
18:01:30 <cverna> morning and happy new year :)
18:02:01 <smooge> ok next up
18:02:13 <smooge> #topic New folks introductions
18:02:14 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
18:02:22 <smooge> I don't expect many new people this week
18:03:40 <smooge> and I am right :)
18:03:55 <smooge> #topic announcements and information
18:03:55 <smooge> #info Bodhi-3.1.0 mostly deployed (backend01 still needs to be upgraded)
18:03:55 <smooge> #info Mass updates/reboots due to Meltdown CVE-2017-5754
18:03:55 <smooge> #info Expect more in the coming time as this gets dealt with
18:03:55 <smooge> #info Happy New Year
18:04:12 <smooge> Any other announcements? We had a 2 week 'break'
18:05:21 <clime> hello
18:05:28 <smooge> hello clime how are you?
18:05:38 <clime> fine, thx, sry for being a bit late
18:05:46 <nirik> oh on bodhi 3.1.0...
18:06:00 <puiterwijk> Bodhi 3.1.0 is fully deployed now actually
18:06:01 <nirik> I reinstalled stg with f27, so 3.1.0 should be running there now.
18:06:07 <smooge> cool
18:06:31 <smooge> #info Bodhi-3.1.0 fully operational. Onward to Alderan
18:06:40 <nirik> oh right, it's in prod... not sure what version is in stg...
18:06:47 <nirik> but I see that it's also not working. ;)
18:07:00 <puiterwijk> Yeah, staging is a very testing release
18:07:11 <pingou> #info mdapi re-installed w/ f27 in stg
18:07:17 <pingou> nirik: it looks good to me there btw :)
18:07:42 <smooge> #info puiterwijk completed operation redeploy proxies to F27
18:08:04 <nirik> pingou: I did prod yesterday too. ;)
18:08:18 <pingou> nirik: cool :)
18:08:22 <pingou> nirik++
18:08:34 <nirik> .title https://bodhi.stg.fedoraproject.org/
18:08:39 <zodbot> nirik: timed out
18:08:43 <smooge> we need zodbot to say "You are out of cookies.. get more cookies for 0.00001 BTC "
18:09:04 <smooge> ok anything else for announcements?
18:09:20 <nirik> I have been trying to make sure we drop all the f25 machiens...
18:09:32 <smooge> Do you have a deadline?
18:09:35 <nirik> there's not too many left
18:09:59 <smooge> like Jan 20?
18:10:06 <nirik> soon yeah...
18:10:25 <pingou> before devconf
18:10:38 <puiterwijk> smooge: December 12, 2017
18:10:40 <smooge> #info Project Annihilate Infrastructure Fedora 25 systems is coming to a close
18:11:04 <puiterwijk> (ideally. We will most likely miss that though, unfortunately)
18:11:11 * smooge watched too many Star Wars movies lately
18:11:22 <nirik> remember: You only have to invent a time machine once. ;)
18:11:55 <smooge> ok in this case...
18:12:07 <smooge> #topic Ticket cleanup
18:12:07 <smooge> #info none this week.
18:12:07 <smooge> #topic Upgrading/Migrating Jenkins
18:12:07 <smooge> #info: Working with Pagure-CI triggers
18:12:08 <smooge> #info A Pagure SCM Plugin for Jenkins
18:12:21 <nirik> bstinson: you around?
18:12:35 <bstinson> yes!
18:12:55 * pingou happy to see this topic :)
18:13:19 <bstinson> so i did some investigation over the holidays, and i think the first thing we need to do is to talk about a couple of minor changes to pagure-ci to support newer jenkins versions
18:13:37 <bstinson> namely we need to add a crumb and a CSRF token to the POST requests when adding webhooks
18:14:06 <bstinson> that will allow the existing workflow in the docs to keep working
18:14:50 <bstinson> i was going to explore a patch this week but suddenly kernel patches...
18:15:03 <nirik> cool. So this is another option? or just a mod to the existing jenkins option?
18:16:33 <bstinson> nirik: this is a change in pagure to change the format of the build triggers slightly
18:16:42 <pingou> bstinson: do you know if older jenkins will also work?
18:16:53 <pingou> or do we need to document a minimum jenkins version supported
18:17:16 <nirik> sure, but now we have: "Type of CI service" and a list with "Jenkins" Do we add a "CentOS CI' there or just leave it at jenkins and just change the urls?
18:17:31 <bstinson> nirik: so there are 2 prongs here
18:17:45 <pingou> nirik: I think the later, since this is a req for newer jenkins anyway
18:17:46 <bstinson> 1.) not breaking existing projects
18:17:49 <bstinson> and
18:18:12 <bstinson> 2.) Onboarding new projects
18:18:31 <nirik> right. and easing migration.
18:19:13 <bstinson> the idea is that we'll stand up a jenkins instance (in the CentOS CI infra) specifically for the miscellaneous projects that don't need their own tenant
18:20:03 <bstinson> to answer pingou's question, we need to break compat for versions less than 2.0 or special-case that in the webhooks
18:20:36 <pingou> bstinson: we could rename the CI services: Jenkins-1.x and Jenkins-2.x
18:20:44 <bstinson> that would work as well
18:21:24 <nirik> RE: our previous topic... our jenkins master is f25. ;) So I want to kill it as soon as we can...
18:22:45 <pingou> ^^
18:23:12 <nirik> so we could just mass move everyone to the new instance and ask them to opt out or request their own instance if they require it?
18:23:28 <bstinson> yep, so standing up an instance is easy, i can probably do the "catch-all" one tonight
18:24:12 <bstinson> that way we have something to look at
18:24:39 <nirik> Sounds good. I guess we will need to tweak urls in pagure side to point to the new instance... along with the changes for 2.0
18:25:09 <smooge> cool. are there any other items on this topic?
18:25:15 <bstinson> yeah, i can hit up pingou maybe tomorrow (depending on how all these reboots go :)
18:25:44 <pingou> bstinson: let's do Monday, I'll be offline tomorrow afternoonw
18:25:50 <bstinson> pingou: ack
18:25:58 <bstinson> smooge: last thing is longer-term integration
18:25:58 <nirik> humm...
18:26:03 <pingou> bstinson: but if you could drop me an email with how you see both system talk, that would be awesome :)
18:26:21 <nirik> so some of these projects don't use the pagure.io CI stuff... they use fedmsgs. I guess we will need to convert them...
18:26:31 <bstinson> pingou: certainly, how do you feel about taking a dep on python-jenkins?
18:26:38 <pingou> bstinson: similarly, if you have a page where the new API is described, maybe I could poke at this Jenkins-2.0
18:26:46 <pingou> bstinson: it's already there :)
18:27:04 <pingou> nirik: but these aren't using our jenkins, do they?
18:27:21 <bstinson> nirik: the catch-all instance can be configured with the existing fedmsg plugins
18:27:44 <nirik> pingou: some are... for example, fedora-comps.
18:28:06 <pingou> oh
18:28:21 <nirik> at least I think it's using fedmsg there
18:28:48 <bstinson> there's also auth, but we can migrate the jobs and get them running while we work on configuring that
18:29:31 <nirik> bstinson: is there some kind of converter for 1.x jobs to 2.x ?
18:30:00 <bstinson> no conversion needed, 2.x will take the configs from 1.x
18:30:09 <nirik> oh nice.
18:30:15 <bstinson> we'll import, then re-save them to make it a little nicer
18:30:17 <pingou> except that most of those are defined in jenkins itself
18:30:36 <bstinson> pingou: jenkins exports config.xml that can be directly imported
18:30:40 <pingou> ok
18:30:45 <pingou> that we have :)
18:31:10 <bstinson> we can also just take the jenkins filesystem and start 2.x over it as a last resort, but that gets a little hairy
18:31:51 <smooge> I would say that sounds like a plan C
18:32:49 <puiterwijk> maybe even a plan F
18:33:09 <nirik> plan Z
18:33:26 <bstinson> heh basically
18:33:37 <nirik> or plan 9 from outer space
18:33:37 <pingou> so what do we need:
18:33:49 <pingou> 1/ a new tenant for all these projects in cico -- bstinson
18:33:57 <pingou> 2/ support for jenkins 2.0+ in pagure -- pingou
18:34:03 <bstinson> 2/ fedora slaves -- bstinson
18:34:38 <pingou> yup good idea :)
18:35:41 <pingou> 4/ a migration plan?
18:35:56 <pingou> how is jenkins going to report back to pagure?
18:35:59 <bstinson> i didn't see any jobs that do non-normal configs in the EL6/EL7 space, are we ok with doing CentOS for those?
18:36:42 <pingou> I think so assuming the package/version don't differ too much
18:38:40 <smooge> ok this all looks good. Anything more on this?
18:39:30 <bstinson> pingou: for reporting back, we'll do the notification plugin (as documented)
18:40:00 <bstinson> the more interesting stuff (fedmsg, conversions to the CI pipeline) can happen later
18:40:26 <pingou> ok cool
18:40:41 <pingou> bstinson: let's sync up on Monday, could you send me a meeting invite?
18:40:46 <bstinson> will do
18:40:53 <pingou> thanks
18:41:34 <bstinson> is there a template to follow for a migration plan?
18:42:06 <bstinson> i can transcribe some of these notes if there's an existing template
18:42:50 <pingou> not really, but we'll need to get things up, migrate a few projects that volunteer (pagure I'm looking at you) and from there announce things and set a date
18:43:06 <smooge> bstinson, do you guys have a template?
18:43:15 <pingou> maybe we could already send an announce for this actually, just stating the will
18:43:41 <bstinson> smooge: i'll gather something up
18:43:58 <smooge> ok that would be helpful on both sides
18:44:12 <bstinson> to the wiki, infra-list, both?
18:44:22 <smooge> Let us go with infra-list starting out
18:44:30 <bstinson> cool
18:44:53 <pingou> after our meeting on Monday, maybe we could also draft a first announcement email :)
18:45:06 <smooge> that sounds good
18:46:27 <smooge> #topic enabling default Pagure fedmsg hooks on Fedora DistGit - clime, pingou
18:46:27 <smooge> #link: https://pagure.io/fedora-infrastructure/issue/6612
18:46:37 <smooge> ok last item I think we can fit in today
18:46:42 <smooge> clime pingou?
18:46:43 <pingou> So we have two git hooks for publishing fedmsg messages
18:46:50 <pingou> one from dist-git and one from pagure
18:46:55 <pingou> both having their own format
18:47:00 <clime> I will let pingou describe it here
18:47:11 <pingou> the dist-git one is installed on all main projects by a daily cron job
18:47:19 <pingou> the pagure one is opt-in from the UI
18:47:41 <pingou> thing is: the pagure hook is sending pagure.git.receive messages and these are not allowed by our policy
18:47:57 <pingou> clime wanted to change that and I wanted to run it by you all before :)
18:48:07 <clime> I was asking pingou today if he things they can be enabled, ye
18:48:15 <clime> *thinks
18:48:19 <nirik> some reduntncy, but fine with me...
18:48:51 <pingou> definitely redundant, but that redundancy is basically opt-in :)
18:49:16 <clime> it simplifies consuming the messages from multiple pagure instances at once...that's the good thing :)
18:49:29 * nirik nods
18:50:29 <pingou> ok, so I'll work on this likely tomorrow if not next week early :)
18:50:39 <puiterwijk> Hmm, if we do that, I'd want to make very sure we blacklist rpms/ etc on pagure.io as namespaces.
18:51:07 <clime> puiterwijk: why is that?
18:51:17 <puiterwijk> The reason being that if we at some point get a script that acts on rpms/$somename messages, and it forgets to check if it comes from src.fp.o, people can trigger it from pagure.io/rpms/$somename
18:51:47 <pingou> puiterwijk: the topic is different
18:51:52 <clime> but the topic should be different for src.fp.o when compared to pagure.io
18:51:57 <pingou> org.fedoraproject.prod vs io.pagure.prod
18:51:59 <clime> ye, pingou faster as usual
18:52:16 <pingou> clime: but typos, typos, typos everywhere :D
18:52:24 <threebean> will fedmsg_meta handle the pagure hook messages correctly?
18:52:40 <pingou> threebean: it should
18:52:49 <puiterwijk> Okay. And we're sure that any script will pick that, rather than what I've also seen as: if '$somesuffix' in topic: <dothis>
18:53:01 <pingou> threebean: https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/blob/develop/fedmsg_meta_fedora_infrastructure/pagure.py#L551
18:53:25 <puiterwijk> I've seen a few rules in fedmsg_meta_fedora_infra that don't read the entire topic, but rather do things like 'compose.complete' in topic
18:53:57 <pingou> puiterwijk: unles they are listening to '*' as topic, they will likely list the topics they want to act on
18:54:00 <threebean> pingou: cool.
18:54:27 <pingou> puiterwijk: they all do, but fedmsg_meta checks this before hand
18:54:41 <pingou> using the __name__ and topic_prefix_re
18:54:47 <puiterwijk> Okay
18:55:02 <puiterwijk> Just wanted to make sure that that's taken into account...
18:55:19 <pingou> fedmsg_meta I fairly confident
18:55:26 <pingou> other scripts... :s
18:55:32 * puiterwijk would hate to find a script that'll rebuild the kernel as soon as $someuser commits to rpms/kernel, only to find it rebuilds it after someone pushes to pagure.io/rpms/kernel...
18:55:41 <puiterwijk> Yeah, I'm afraid for the other scripts...
18:55:53 <puiterwijk> Which is why I would much rather just stay safe and just blacklist it for now.
18:56:23 <pingou> it's easily doable anyway
18:56:26 <puiterwijk> But if you say it's sufficiently covered, sure.
18:57:03 <pingou> BLACKLISTED_GROUPS = ['forks', 'group']
18:57:17 <puiterwijk> Yeah
18:57:19 <pingou> -BLACKLISTED_GROUPS = ['forks', 'group']
18:57:21 <pingou> +BLACKLISTED_GROUPS = ['forks', 'group', 'rpms']
18:57:30 <puiterwijk> pingou: +containers, +tests, etc :)
18:57:50 <smooge> ... at what point does a white list make sense :)
18:57:51 <puiterwijk> And I'm not saying that that blacklist should stay there permanently, or that we must do that. I just want it as long as we're not 100% sure
18:58:07 <puiterwijk> smooge: the list of things to blacklist is small :)
18:58:16 <clime> I am not liking the blacklist personally
18:58:18 <puiterwijk> I think it's like 4 or 5?
18:58:42 <pingou> rpms, modules, container, tests
18:58:43 <puiterwijk> clime: why not, and how would you then alleviate my concern, given that the "X in topic" method is used by people?
18:59:25 <clime> but it doesn't really depend on me. Well, thing is when you introduce the blacklist, it then stays there like permanently
18:59:27 <puiterwijk> clime: it just means we can't have a group called "rpms" in pagure.io... what's the deal?
18:59:29 <nirik> I guess without a blacklist it just becomes that script authors problem. ;)
18:59:36 <puiterwijk> clime: doesn't have to be permanent
18:59:54 <puiterwijk> clime: I'm just saying I want this until we have another way to assure that it doesn't happen
19:00:00 * nirik is fine blacklisting those namespaces. I think they might be confusing anyhow.
19:00:04 <clime> well, not sure what will be the point when someone decides to clean the problems in the scripts
19:00:18 <clime> just mentioning...
19:01:50 <smooge> ok I don't see us moving on this at the moment.
19:02:03 <smooge> and I think there is another meeting starting (now/soon?)
19:02:19 <timc> Hi all - quick introduction as I know we're running out of time - Interested in helping out with sysadmin tasks and joining apprentice program. FAS username: timjcasey.  Have been an Ambassador for quite some time now, will be updating my wiki page later today with more info on my background etc.
19:02:31 <smooge> hi timc
19:02:37 <nirik> welcome timc
19:02:38 <timc> sorry if I missed the intro section at the start :)    Was a bit late.
19:02:50 <timc> its 6am here - hard to wake up so early :D
19:03:17 <cverna> welcome timc
19:03:27 <timc> thanks all
19:04:02 <pingou> welcome timc :)
19:04:12 <smooge> #topic Open Floor
19:04:22 <pingou> so I think for now we can enable that topic, and blacklist these groups
19:04:24 <clime> mainly you always need thing about the blacklist when you add a new namespace at src.fp.o...
19:04:25 <pingou> and reconsider later :)
19:04:28 <smooge> timc, please join #fedora-admin after this and we can set up the account items
19:04:31 <clime> sry for late argument
19:04:57 <smooge> ok lets take the blacklist/whitelist/greylist to infrastructure or proper list
19:05:00 <timc> smooge: no probs, already there
19:05:08 <smooge> are there any open floor items?
19:06:21 * pingou has none
19:06:32 <smooge> ok in that case... thank you all for coming.
19:06:41 <nirik> thanks smooge and everyone
19:06:44 <smooge> #endmeeting