18:00:14 <nirik> #startmeeting Infrastructure (2016-09-15)
18:00:14 <zodbot> Meeting started Thu Sep 15 18:00:14 2016 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:14 <zodbot> The meeting name has been set to 'infrastructure_(2016-09-15)'
18:00:14 <nirik> #meetingname infrastructure
18:00:14 <zodbot> The meeting name has been set to 'infrastructure'
18:00:14 <nirik> #topic aloha
18:00:14 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:14 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:14 <nirik> #topic New folks introductions
18:00:26 <smooge> .hello
18:00:26 <zodbot> smooge: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
18:00:27 <puiterwijk> .hello puiterwijk
18:00:28 <trishnag> .hello trishnag
18:00:28 <zodbot> puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' <puiterwijk@redhat.com>
18:00:31 <zodbot> trishnag: trishnag 'Trishna Guha' <trishnaguha17@gmail.com>
18:01:03 <nirik> morning everyone
18:01:17 <trishnag> Good morning :)
18:01:48 <dgilmore> hi
18:01:52 <nirik> any new folks today that would like to give a short one line introduction of themselves?
18:01:55 <marc84> hi
18:02:50 * cverna here
18:04:09 <maxamillion> .hello maxamillion
18:04:12 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
18:04:13 <nirik> ok, I guess lets go on to status/info
18:04:27 <nirik> #topic announcements and information
18:04:27 <nirik> #info bodhi stakeholders meeting to sync up on a bunch of things - bowlofeggs
18:04:27 <nirik> #info inactive apprentices dropped - kevin
18:04:27 <nirik> #info new sigul in production - patrick
18:04:28 <nirik> #info reinstalled sign-vault03 and upgraded secondary-vault01 - kevin/patrick
18:04:29 <nirik> #info new koji tags for infrastructure packages (epel7, f25, f24) - patrick
18:04:30 <nirik> #info new wiki in staging using openid - kevin / patrick
18:04:32 <nirik> #info bodhi-backend01.stg moved to f24 - kevin
18:04:34 <nirik> #info bodhi-backend01 and autosign01 removed - kevin
18:04:36 <nirik> #info Next years budget time is starting up. If there are major projects needing hardware, please let smooge know.
18:04:39 <nirik> #info New bodhi beta in stg - please test before monday (or else!) - bowlofeggs
18:04:41 <nirik> #info New fedimg in prod, based on compose - sayan
18:04:48 <nirik> anything anyone wants to discuss further from those? or add to/note?
18:04:52 <sayan> .hello sayanchowdhury
18:04:53 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
18:04:54 <bowlofeggs> .hello bowlofeggs
18:04:56 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:06:15 <dgilmore> nirik: epel6 epel7 f23 f24 f25
18:06:42 <nirik> hum?
18:06:46 <nirik> oh, for the infra koji tags?
18:07:14 <kushal> .hellomynameis kushal
18:07:15 <zodbot> kushal: kushal 'Kushal Das' <mail@kushaldas.in>
18:07:18 <kushal> .hellomynameis kushal
18:07:20 <zodbot> kushal: kushal 'Kushal Das' <mail@kushaldas.in>
18:08:02 <aikidouke> .hello aikidouke
18:08:03 <zodbot> aikidouke: aikidouke 'Zach Villers' <zachvatwork@gmail.com>
18:09:03 <nirik> ok, if nothing else will go on to discussion items...
18:09:06 <nirik> #topic Freeze coming up soon. What to get done before then? - kevin
18:09:10 <dgilmore> nirik: yep
18:09:32 <nirik> so, we have a number of things in the air... freeze is coming up on the 27th
18:09:42 <bowlofeggs> nirik: how long do freezes typically last?
18:10:06 <nirik> bowlofeggs: until the release for that milestone... so the one one that starts on the 27th is for Fedora 25 beta
18:10:24 <bowlofeggs> bodhi 2.2.0 probably won't make it into epel7 stable before the 27th (which is fine) so i guess we'll wait till after the freeze for that
18:10:27 <nirik> 2016-10-11 is beta release
18:10:31 <nirik> unless it slips
18:10:42 <bowlofeggs> that's ok from where i sit
18:10:49 <nirik> well, we can bypass that if you want to land it before freeze...
18:11:13 <bowlofeggs> i don't have a need to bypass
18:11:17 <nirik> as long as it's enough before freeze so we can fix things. :)
18:11:21 <bowlofeggs> but if releng wants to get any of those changes in i'm happy to
18:11:40 <nirik> it would be nice to get synced up so I'd kinda prefer to just get it landed.
18:11:53 <bowlofeggs> we can do that
18:12:02 <bowlofeggs> i guess we would just tag it into the infra repo?
18:12:18 <nirik> yeah
18:12:28 <bowlofeggs> well i'm hoping to release it next week
18:12:35 <nirik> sounds fine to me.
18:12:42 <bowlofeggs> so that would give me ~1w to address problems
18:12:45 <nirik> lets see... other stuff we have pending:
18:13:02 <nirik> * wiki: we have staging setup with the new one except for some outstanding issues:
18:13:13 <nirik> - theme work ( ryanlerch is working on that)
18:13:32 <nirik> - porting the mwclient python stuff to handle logins with openid (no one is commited to this yet)
18:13:46 <nirik> - migrating db to postgres
18:14:07 <smooge> I was looking to build out a set of proxy and mirrormanager boxes in RDU02
18:14:10 <puiterwijk> I would personally vote -1 to blocking on the migrate to postgres for the upgrade. As said, I would suggest splitting it
18:14:48 <nirik> smooge: that would be a good thing to get done yes... you want it for your list, feel free to take it. ;)
18:14:58 <bowlofeggs> postgres++
18:15:00 <nirik> puiterwijk: ok, fair enough, we do need openid handling tho
18:15:11 <puiterwijk> nirik: yes. I have started on that this morning
18:15:24 <bowlofeggs> oh i wasn't disagreeing with puiterwijk, i just really like postgres and wanted to ++ it ☺
18:15:32 <nb> .hello nb
18:15:33 <zodbot> nb: nb 'Nick Bebout' <nb@nb.zone>
18:15:40 <nirik> oh, cool. So, whats the chance that it and the theme will be done next week? or should we plan to do this after the freeze?
18:15:44 <puiterwijk> And will hopefully have that very soon, given I acn reuse a lot of the code from python-fedora
18:16:40 <nirik> ok. I can check with ryan about the theme and we can just see where we are next week?
18:16:51 <puiterwijk> Yep
18:17:11 <nirik> https://stg.fedoraproject.org/wiki/ is the staging instance if anyone wants to poke at it and check to make sure everything is working
18:17:46 <nirik> #info wiki upgrade: will check on status of theme and openid script auth next week and decide if we want to land it before the freeze or after.
18:18:13 <nirik> * hyperkitty / mailman3 upgrade: I think abompard is going to try and do that tomorrow morning or early next week...
18:18:19 <nirik> this includes the social login stuff.
18:18:42 <nirik> #info hyperkitty / mailman3 upgrade with social media logins - hopefully tomorrow or early next week.
18:18:52 <nirik> relrod: you around? whats the status on modernpaste?
18:18:58 <bowlofeggs> social local mobile
18:20:06 <nirik> so we have modernpaste in stg now, but we need to coordinate updating prod with fpaste changes and also there were some features we wanted to add.
18:20:23 <nirik> https://modernpaste.stg.fedoraproject.org/ is that site if anyone wants to look at it.
18:21:16 <nirik> I can ask relrod status when hes next around.
18:21:31 <nirik> #info modernpaste: will find out current status and see when we want to land it.
18:22:06 <nirik> ok, next, I am thinking of migrating fedora-infrastructure trac over to pagure...
18:22:16 <nirik> possibly tomorrow if there's no objections. ;)
18:22:33 <nirik> I'm thinking since our tags are such a mess that I will import it without tags
18:23:03 <nirik> Can pagure projects have multiple git repos? or only one?
18:24:02 <puiterwijk> only one
18:24:08 <puiterwijk> a project == a repo
18:24:09 <nirik> I guess we will mess with the ansible repo later as a seperate project
18:24:24 <puiterwijk> nirik: well... a project actually has three repos, but one code repo
18:24:35 <puiterwijk> (code repo, tickets repo and docs repo)
18:24:51 <nirik> right. But I wanted to have the current git repo for fedora-infra and also our ansible repo (if we can figure out how to sync it)
18:25:03 <puiterwijk> Right. That would be two different projects
18:25:12 <puiterwijk> and syncing should come natively to Pagure at some potint
18:25:13 <puiterwijk> point
18:25:18 <nirik> sure, but a bit confusing.
18:25:38 <nirik> "I want to submit a PR for fedora infra ansible" and they may not find the right place
18:25:44 <puiterwijk> not saying I disagree :)
18:26:00 <jflory7> .hello jflory7
18:26:04 <zodbot> jflory7: jflory7 'Justin W. Flory' <jflory7@gmail.com>
18:26:24 <nirik> anyhow, I guess I will move over what we have and we can figure out ansible repo later.
18:26:44 <nirik> #info nirik will migrate fedora-infrastructure trac over to pagure tomorrow
18:26:53 <maxamillion> nirik++
18:27:01 <nirik> any other things we have in flight that we should see if we can land before freeze?
18:27:05 <jflory7> nirik++
18:27:09 * nirik thinks there were some more but can't think of them now.
18:27:11 <jflory7> That's awesome news :)
18:27:16 <puiterwijk> nirik: autosigning will hopefully live before freeze
18:27:20 <puiterwijk> at least for rawhide
18:27:43 <nirik> yes, that will be great. :) I have wanted that for a long long time.
18:28:16 <puiterwijk> Oh, also
18:28:25 <puiterwijk> The first repos to be autosigned will be the new infra tags
18:28:35 <puiterwijk> So those will hopefully replace the infra repos very soon
18:28:36 <nirik> #info autosigning should be setup before freeze.
18:28:57 <nirik> puiterwijk: we need to check and see what package(s) need to be rebuilt.
18:29:11 <puiterwijk> nirik: yep. Was going to do that when we have it enabled and working
18:30:04 <nirik> yeah.
18:31:11 <nirik> ok, I can't think of other stuff, but there may be some. ;)
18:31:30 <nirik> So, if anyone sees something we said we would land outside freeze or the like, please do let us know.
18:31:43 <puiterwijk> Oh, new Ipsilon release should make its way to prod very soon, please test in staging.
18:31:45 <nirik> #topic mass update/reboot cycle next week?
18:31:48 <puiterwijk> This fixes unicode support :)
18:32:02 <nirik> and before freeze, I was thinking of an update/reboot cycle next week.
18:32:04 <puiterwijk> nirik: +1 on reboot cycle
18:32:08 <nirik> puiterwijk: cool. :)
18:32:28 <nirik> so I was thinking cloud monday, build tuesday and rest wed...
18:32:49 <nirik> tflink / pbrobinson: mass update/reboot ok with you next week?
18:33:25 <puiterwijk> I will schedule PTO on monday and tuesday then. :-)
18:33:34 <nirik> will check with them out of meeting and send announcements.
18:33:36 <nirik> ha. ;)
18:33:57 <pbrobinson> nirik: yup
18:34:09 <nirik> cool.
18:34:29 <nirik> #info update/reboot cycle next week mon(cloud), tuesday(build), wed (everything else).
18:34:33 <pbrobinson> nirik: just co-ordinate obviously but next week it reasonable
18:34:50 <nirik> will do
18:34:52 <nirik> #topic Apprentice Open office hours
18:35:03 <nirik> any apprentices with questions or looking for things to work on ?
18:35:56 <nirik> ok.
18:36:04 <tflink> nirik: shouldn't be a problem
18:36:18 <nirik> FYI, I will keep the easyfix tag when we move to pagure... so all that should stay the same. Might even be able to make some more marked that way
18:36:35 <nirik> tflink: great
18:36:53 <nirik> So, there's a few tickets maxamillion filed we could discuss real quick...
18:37:12 <maxamillion> yeah, I was going to bring those up in Open Floor
18:37:18 <nirik> #topic Releng-automation
18:37:22 <maxamillion> I didn't know how meeting items get added to the agenda
18:37:24 <nirik> .ticket 5466
18:37:26 <zodbot> nirik: #5466 (Fedora RelEng Automation) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5466
18:38:00 <nirik> I agree with pingou there... if we can just reuse what we have now...
18:38:09 <maxamillion> that won't work
18:38:20 <nirik> ok
18:38:21 <maxamillion> I'm sorry, I didn't notice the reply  ... I would have added info
18:38:30 <maxamillion> we need to be able to pass information from the fedmsg into the playbook
18:39:02 <maxamillion> I suppose we could work around that though
18:39:23 <maxamillion> actually, no ... the idea I had would ahve issues
18:39:31 <maxamillion> have*
18:39:34 <nirik> so can you give some examples of what this would be used for?
18:40:00 <maxamillion> automatically releasing Atomic Host every two weeks and/or releasing Docker images every two weeks
18:40:35 <maxamillion> so the fedmsg would come in with the data, and the playbook would need that information to operate on the correct compose or docker image tag
18:40:42 <maxamillion> well ... name and tag
18:40:56 <maxamillion> those are the two use cases I'm trying to focus on now, I'm sure there will be others as time goes on
18:41:35 <nirik> well, those both might be possible in our current setup.
18:41:51 <nirik> we currenly have a fedmsg listener on batcave01.
18:42:20 <nirik> when it gets a fedmsg from fas about ssh keys changing it looks at the message and sees who the user was.
18:42:48 <nirik> and depending it fires off fas to update on a subset of our hosts...
18:43:02 <maxamillion> nirik: right, but that means that we'd need to write logic into the listener for each scenario in which we need to extract data from a fedmsg and pass it to a playbook, right?
18:43:17 <nirik> yeah.
18:43:48 <nirik> which I guess could be a pain... but also...
18:44:06 <nirik> this is running as root on everything
18:44:47 <nirik> but perhaps we could make it more generic with some checks
18:45:04 <maxamillion> I'm not sure we need or necessarily want root for releng automated tasks
18:45:37 <puiterwijk> Well, I'm pretty sure that for running ansible we'll need to run ansible-playbook as root
18:45:41 <nirik> well, if we don't re-use what we have now, it means setting up an entirely parallel infra for this
18:45:49 <maxamillion> puiterwijk: why?
18:46:02 <maxamillion> nirik: fair
18:46:02 <puiterwijk> maxamillion: because ansible needs to call out to external hosts. For which it needs the root key?
18:46:03 <nirik> maxamillion: it needs to talk to the agent that has roots ssh key
18:46:30 <maxamillion> puiterwijk: in the ticket I mention creating a new user on the releng hosts for automated tasks with it's own key
18:48:01 <maxamillion> I'll just sort this out using the fedmsg listener already in place instead of trying to introduce something new, it seems like nobody likes the idea that I proposed and I don't want to go against the grain ... I'm not married to my approach, it just seemed simple and generic enough to be used for almost anything
18:48:38 <nirik> I'm not against adjusting our current setup to be more flexable (as long as it's still secure).
18:48:51 <nirik> I just don't want to make 2 of everything if we can avoid it.
18:49:06 <maxamillion> how would I go about getting the releng automation ansible git repo into the fedora infra? it's in pagure now, what criteria need to be met so that it's contents can be executed by the fedmsg listener?
18:49:36 <nirik> well, it would have to be on batcave01... either pulled from pagure or synced or moved there or something
18:49:56 <maxamillion> alright, I'll sync with someone on that later then
18:50:00 <nirik> Is the idea of it being there so more folks can contribute?
18:50:01 <maxamillion> we can move on if everyone's good
18:50:07 <maxamillion> nirik: yes
18:50:16 <maxamillion> nirik: pull requests and such
18:50:16 <nirik> or could it just be in infra ansible?
18:50:36 <nirik> yeah, ok. Note that we plan to make our ansible repo available for that kind of thing too... we just need pagure to get sync ability...
18:51:19 <nirik> let me ponder on this and see if we can make our setup for flexable for you...
18:51:22 <maxamillion> nirik: I'm alright with it living inside of fedora infra ansible once that's moved to pagure
18:51:30 <nirik> ok, cool.
18:51:40 <maxamillion> nirik: I'll look at it, I don't mind doing the work on the fedmsg listener
18:51:43 <nirik> anything else on this? did you have another ticket?
18:51:51 <maxamillion> I have two other tickets :)
18:52:01 <nirik> ok. ;) what one next?
18:52:22 <maxamillion> lets go with 5467
18:52:27 <nirik> .ticket 5467
18:52:27 <maxamillion> https://fedorahosted.org/fedora-infrastructure/ticket/5467
18:52:28 <zodbot> nirik: #5467 (Providing a "latest" endpoint for Fedora Atomic Host's release artifacts) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5467
18:52:40 <nirik> #topic latest link for fedora atomic host
18:52:52 <nirik> so, I guess compose doesn't want to provide this?
18:52:54 <maxamillion> this is basically just a "how do I do this?" request because I'm out of my depth
18:53:02 <nirik> well, there's 0 good way. ;)
18:53:07 <maxamillion> yeah :/
18:53:21 <nirik> My next thought was to try and come up with a apache redirect/rewrite rule... but that could be very ugly
18:53:36 <nirik> I really really want to avoid changing proxy config every 2 weeks for this
18:53:37 <maxamillion> oh yeah, it would be extremely painful
18:54:03 <nirik> because if something automated there messes up, we are in a world of downtime.
18:54:24 <maxamillion> what if we got adamw's 'latest' page thing into Fedora Infra as a web app somewhere and then slap this in there? it's basically the same target audience
18:54:51 <maxamillion> this maybe? https://www.happyassassin.net/nightlies.html
18:54:59 <nirik> we could...
18:55:12 <adamw> maxamillion: that would kinda imply reconciling how nightlies decides images are 'good' with how the current two-week nomination script does that
18:55:17 * nirik points to https://nightly.fedoraproject.org
18:55:25 <nirik> (which is a redirect to that :)
18:55:29 <adamw> but i am all in favour of doing that and making it all nice and official
18:55:31 <maxamillion> adamw: we could write a small app in flask :)
18:55:34 <adamw> nirik: oh neat, who set that up?
18:55:56 <nirik> I did. It used to have some old crap about livecd's and was broken, so when I removed it I just pointed it to you. :)
18:56:07 * adamw is quite proud of nightlies explicitly *not* being a webapp, but hey =)
18:56:11 <adamw> (it's a static page generator, basically)
18:56:34 <maxamillion> adamw: well, we could just have a fedmsg listener that kicks the regeneration of the page or something?
18:56:37 <adamw> but yeah, broadly i always wanted to make nightlies more official-ish, i'm happy to work with whoever to figure out a plan.
18:56:40 <nirik> yeah, for that we could run some generation script on sundries01 and sync it out to proxies
18:56:43 <adamw> maxamillion: that's exactly how it works. ;)
18:56:48 <maxamillion> adamw: YESSSSSSS
18:57:03 <maxamillion> alright, I'll follow up with that approach then
18:57:15 <nirik> then it's just static pages on proxies, and hopefully unlikely to break anything else. ;)
18:57:27 <maxamillion> nirik: +1
18:57:29 <adamw> maxamillion: https://pagure.io/fedora_nightlies/blob/master/f/fedora_nightlies.py#_693
18:57:45 <maxamillion> adamw: this is fantastic
18:57:56 <adamw> maxamillion: autocloud has changed a bit since i wrote it; it still works but it's possible we could simplify things a bit now. i've had https://sayanchowdhury.dgplug.org/autocloud-whats-new/ open in a tab forever as a reminder to work on that
18:57:59 <maxamillion> I'm happy to move onto the next ticket if everyone is good
18:58:08 <maxamillion> adamw: +1
18:58:17 <adamw> maxamillion: shoot me an email and subscribe me to the ticket or something, we'll figure it out
18:58:24 <nirik> that approach sounds fine to me. I'm happy to help whoever works on implementing it.
18:58:49 <maxamillion> adamw: +1
18:58:52 <nirik> maxamillion: what was your last ticket? (in 1minute left. ;)
18:59:06 <maxamillion> https://fedorahosted.org/fedora-infrastructure/ticket/5476
18:59:12 <nirik> .ticket 5476
18:59:13 <zodbot> nirik: #5476 (How to bring an external ansible repo into Fedora Infra) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/5476
18:59:20 <nirik> #topic External ansible repo
18:59:41 <nirik> yeah, I was pondering on this one.
19:00:26 <maxamillion> long story short, I would *really* like to be able to "bring in" a point in time snapshot of the openshift-ansible repo and re-rebase on it periodically because I don't think I can keep up with them otherwise but they call things very generic names so it would need to be split out enough that it wouldn't conflict with anything else because 'master' and 'node' can easily conflict
19:00:27 <nirik> How about we check out a specific hash of them on batcave, but not in the ansible repo... but we will need to sort out how to make sure they are callable by regular ansible
19:00:48 <nirik> which I think we could do with role paths, etc.
19:01:12 <maxamillion> nirik: alright, just so we don't run over time here ... what's the best course of action to follow up on this?
19:01:48 <nirik> I can make a commit to do this... do we have already some subset of those repos in?
19:02:00 <nirik> so, it would take I think:
19:02:40 <nirik> add git checkouts to batcave role, add ansible config for role-path or whatever, and adjusting our existing setup to use the new checkouts instead of a copy inside our ansible repo
19:03:02 <maxamillion> nirik: we don't currently ... we did for a short period a very long time ago but it's since been ripped out and everything in currently I either wrote or based off the OSBS team's stuff, problem there is it doesn't take into account all the complicated clustering stuff needed for multi-node deployments
19:03:20 <nirik> yeah, so we would need to re-port back to their stuff right?
19:03:38 <nirik> hum, but there might still be conflicts if we just import their roles...
19:04:14 <nirik> we may have to play with it some and see what will work here.
19:04:21 <maxamillion> nirik: no, this will be a rip and replace ... I have to throw away everything I've done so far to pivot to multi-node deployments (which is something I didn't want to do, but it was a reality of circumstance)
19:05:01 <nirik> sure, but I mean even if we check out their repos seperately, if we call them from our repo we may run into problems with generic variables, etc.
19:05:20 <nirik> would have to see.
19:05:26 <nirik> is this something you need before freeze?
19:05:50 <maxamillion> nirik: it is not, it's something I probably won't even start working on seriously for a few weeks
19:06:07 <maxamillion> nirik: there's a lot up in the air right now as it is and that's not super urgent right now
19:06:37 <nirik> ok. Ping me out of meeting someday and we can see what works.
19:06:51 <nirik> ideally I think checking them out seperately and just using them for these would be best
19:07:04 <nirik> as long as their isn't any overlapping variables or the like
19:07:29 <maxamillion> +1
19:07:46 <nirik> maxamillion: anything else?
19:07:59 <maxamillion> nirik: nope, that's all from me for today :)
19:08:05 <maxamillion> nirik: thank you
19:08:05 <nirik> #topic Open Floor
19:08:09 <nirik> anything for open floor?
19:08:24 <nirik> thoughts, comments, favorate pumpkin spice product?
19:09:21 <smooge> I like muffins
19:09:39 <nirik> I have some pumpkin spice coffee. ;)
19:09:40 <maxamillion> Kind bars pumpkin spice offering is better than expected but I feel super basic buying it :)
19:09:55 <smooge> the pumpkin spice toilet paper is not recommended
19:10:00 <nirik> anyhow, thanks for coming everyone!
19:10:06 <nirik> #endmeeting