18:00:24 <nirik> #startmeeting Infrastructure (2017-02-16)
18:00:24 <zodbot> Meeting started Thu Feb 16 18:00:24 2017 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:24 <zodbot> The meeting name has been set to 'infrastructure_(2017-02-16)'
18:00:24 <nirik> #meetingname infrastructure
18:00:24 <zodbot> The meeting name has been set to 'infrastructure'
18:00:24 <nirik> #topic aloha
18:00:24 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:24 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:24 <nirik> #topic New folks introductions
18:00:33 <athos> hello :)
18:00:35 <clime> hello
18:00:37 <marc84> hi everyone
18:00:39 <smooge> hello
18:00:39 <bowlofeggs> .hello bowlofeggs
18:00:40 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:00:47 <jcline> .hello jcline
18:00:48 <zodbot> jcline: jcline 'Jeremy Cline' <jeremy@jcline.org>
18:00:53 * relrod here
18:01:16 <trishnag> .hello trishnag
18:01:18 <zodbot> trishnag: trishnag 'Trishna Guha' <trishnaguha17@gmail.com>
18:01:25 <nirik> morning everybody
18:01:40 <tflink> .hello tflink
18:01:43 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
18:01:48 <puiterwijk> hello
18:02:10 <wfturnage> Hey All. I'm new :D
18:02:16 * threebean waves
18:02:32 <nirik> Any new folks like to give a short one line introduction of themselves?
18:02:35 <nirik> wfturnage: ^
18:03:11 <wfturnage> Sent my "hello world" email but it doesn't look like it came through. I have 6ish years of sysadmin experience
18:03:28 <wfturnage> mostly just OS work on RHEL 4 - 7
18:05:31 <nirik> welcome!
18:05:37 <wfturnage> thank you!
18:05:47 <nirik> I assume you're more interested in sysadmin stuff? or application devel? or both?
18:06:12 <wfturnage> both, but my strength is sysadmin work.
18:06:41 <nirik> cool. See #fedora-admin after the meeting and we can get you added to the apprentice group, etc....
18:06:46 <nirik> any other new folks?
18:06:51 <wfturnage> I do have a question about the mailing list. I see my email in the archives but I didnt see it come through to my client
18:06:58 <wfturnage> will do
18:07:09 <nirik> are you by chance using google/gmail?
18:07:17 <wfturnage> i am on gmail
18:07:20 * pingou late
18:07:29 <relrod> pingou: tsk!
18:07:33 <nirik> yeah, thats a 'feature'. It doesn't show you emails from yourself. :)
18:07:56 <wfturnage> hah! thanks for clearing that up for me.
18:08:08 <nirik> many many many people have hit that.
18:08:12 <nirik> #topic announcements and information
18:08:12 <nirik> #info CAA dns records added to fedoraproject.org and getfedora.org - kevin
18:08:12 <nirik> #info DKIM mail signing active for pagure.io/fedoraproject.org/lists - kevin
18:08:12 <nirik> #info mirrorlist containers now active on all proxies - kevin/patrick
18:08:13 <nirik> #info mass rebuild in rawhide finished sunday - kevin
18:08:14 <nirik> #info the-new-hotness 0.8.0 is now in production - jcline
18:08:17 <nirik> #info anitya 0.11.0 is now in production - jcline
18:08:19 <nirik> #info infra SOPs migrated to https://pagure.io/infra-docs/ today - jcline
18:08:24 <threebean> jcline++ nice!
18:08:26 <zodbot> threebean: Karma for jcline changed to 12 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:08:27 <nirik> any status/info/announcements folks would like to add to the above? or discuss from there?
18:08:55 <relrod> #info modern-paste in prod now
18:08:57 <nirik> jcline: we need to setup batcave01 to just pull those from there now.. to avoid multiple places people might commit
18:09:20 <jcline> Yeah, I have a little script I was going to run by people to pull
18:09:20 <clime> #info patch for use of new dist-git package on staging has been pushed
18:09:28 <threebean> #info still working out kinks in MBS in staging.  creds are set.  next up is bootstrapping data.
18:09:30 <jcline> And then we just set the mirror=true flag on /git/infra-docs
18:10:07 <jcline> (the script is just "pushd /git/infra-docs && git fetch origin")
18:10:15 <clime> I would need somebody to run the pkgs playbook on pkgs staging so that I can start testing it.
18:10:47 <nirik> clime: I can. which host? pkgs02.stg?
18:10:53 <nirik> jcline: sounds good to me.
18:11:18 <jcline> I didn't see anything in the batcave ansible role to set up the git repo itself. I could add that, though.
18:11:39 <nirik> yeah, I think we just treated it as 'data we already have'
18:11:56 <bowlofeggs> #info i'm trying to get bodhi 2.4.0 deployed today
18:12:02 <bowlofeggs> i've hit a bunch of snags
18:12:09 <bowlofeggs> the last thing i need is builds to be tagged ☺
18:12:10 <clime> nirik: thank you, for start pkgs02 but I would like to deploy in pkgs01 also as there are some host related settings in the playbook
18:12:48 <clime> nirik: namely role distgit/pagure is not installed on pkgs02
18:12:52 <nirik> clime: ok, running. I hope that won't mess up any of pingou's pagure testing. ;)
18:13:09 <nirik> bowlofeggs: anything we can help with? or just stuff that needs slogging thru?
18:13:29 <pingou> clime: oh yes, sorry I didn't run that playbook today :(
18:13:30 <bowlofeggs> nirik: i wrote in #fedora-admin just moments ago - i just need three builds tagged into the infra repo
18:13:39 <bowlofeggs> bodhi-2.4.0-1.fc25 bodhi-2.4.0-1.fc24 and bodhi-2.4.0-1.el7
18:13:39 <clime> pingou: nevermind :)
18:13:43 <pingou> saw the ping, then went onto something else and forgot :s
18:14:02 <clime> I made some patches for mock instead :)
18:14:11 <pingou> oh boy
18:14:19 <bowlofeggs> nirik: since there are already pushes in progress, how is a thursday night deployment? too close to friday, or ok?
18:14:28 <pingou> clime: ping me twice next time, I really didn't mean to put you in that situation :S
18:14:40 <clime> pingou: ok :D
18:15:14 <nirik> bowlofeggs: tagged. Fine with me, but check with mboddu (who is on push duty this week)
18:15:15 <bowlofeggs> (the f24 one is only needed because staging backend is still f24)
18:15:21 <bowlofeggs> nirik: will do, thanks!
18:15:32 <nirik> yeah, I was meaning to get to that, but kept getting sidetracked. ;)
18:16:22 <nirik> ok, any other status/info/blockages? :)
18:17:20 <nirik> ok, moving on then...
18:17:23 <nirik> #topic fedorahosted migration status - kevin
18:17:49 <nirik> so, I went and looked at at least the top 20 projects... almost everyone is migrated or in progress.
18:17:57 <pingou> \ó/
18:18:08 <nirik> I mailed a few folks and got a few replies where they had moved, but didn't note it on the hosted side
18:18:08 <smooge> cool
18:18:18 <smooge> not so cool
18:18:28 <nirik> There's more to followup on I am sure. I updated the wiki page some.
18:19:12 <nirik> we are down to less than 2 weeks left.
18:19:15 <smooge> idea to make it more visible.. make it read only from now on :)
18:19:38 <pingou> lo
18:19:40 <pingou> l
18:19:48 <nirik> I'm still ok with adding a banner and a note to the ssh hook. ;) ro might be a bit much
18:20:17 <smooge> ok well don't let it be said I don't go for bold solutions :)
18:20:47 <nirik> 😁
18:21:09 <nirik> anyhow, I guess I don't have anything further... will keep bugging people. Everyone should bug any projects they know...
18:21:39 <nirik> #info less than 2 weeks left. MIGRATE YOUR FEDORAHOSTED PROJECTS NOW!!!!
18:21:57 <nirik> #topic requiring blogs to be https - kevin
18:22:12 <nirik> so, I posted this to the list... and got a fair number of replies on it. :)
18:22:30 * pingou is among the guilty non-https blogs
18:22:55 <nirik> I'm afraid there will be a number of folks in bexelbie's situation... their provider just doesn't support it currently
18:23:36 <nirik> so we could take a middle ground... urge everyone to move to it and try and get as many as we can, then revisit down the road later in the year.
18:24:33 <nirik> anyhow, we can continue that on the list... unless someone wants to chime in here.
18:25:25 <nirik> ok, any other discussion items?
18:26:48 * jcline has nothing
18:27:00 <nirik> #topic Apprentice Open office hours
18:27:10 <nirik> any apprentices with questions/comments/ideas/etc?
18:28:55 * nirik listens to the silence. :)
18:29:03 <nirik> ok, I have a learn about today...
18:29:09 <nirik> #topic Learn about: mirrorlist containers - kevin
18:29:55 <nirik> so, we have a application called mirrorlist. This is a small wsgi app that takes a pile of data from mirrormanager and serves it to clients.
18:30:17 <nirik> when you use dnf or yum to request a metalink or mirrorlist from mirrors.fedoraproject.org it's the thing that gives you that data.
18:30:41 <nirik> mirrors.fedoraproject.org is pointing to all our proxy servers. So, they get the requests first.
18:31:27 <nirik> then in the past what we did is those proxies would use haproxy in front of a bunch of mirrorlist servers and send the request out over the vpn to our main datacenter then out to the mirrorlist and back then reply to the client
18:31:52 <nirik> This meant that if the vpn was down or slow or whatever, things would stop working.
18:32:09 <nirik> it also meant a ton of traffic going over our vpn for these requests (which are a large number)
18:32:22 <nirik> So, recently, we replaced this setup.
18:32:53 <nirik> Now, we have each proxy is running 2 mirrorlist docker containers (built in our OSBS build system and using f25 base)
18:33:33 <nirik> each hour new data comes from mirrormanager, then a script starts the backup container, makes sure it's ok, then starts the primary container with the new data and makes sure it's ok.
18:33:54 <nirik> then it disables the backup one again. So, there's 0 times something is not answering requests.
18:34:10 <nirik> and if bad data comes out, the backup one keeps running and tells us the data is bad
18:34:20 <nirik> And now if the vpn is down, it doesn't matter.
18:34:33 <nirik> and it's hopefully made requests also much faster
18:35:16 <jcline> \o/
18:35:21 <nirik> lets see... oh, right now we have all the old mirrorlist servers as backup, but I will probibly remove most of them next week
18:35:56 <nirik> Here's a graph of traffic on our vpn server. See if you can tell when we rolled these out: https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=bastion01.phx2.fedoraproject.org;plugin=interface;plugin_instance=eth0;type=if_octets;begin=-604800
18:36:15 <bowlofeggs> nirik: pretty neat!
18:36:31 <bowlofeggs> nirik: i wonder if openshift could be used to help make that go in the future too?
18:36:34 <nirik> yeah, this really seemed like a good fit for containers
18:36:38 <nirik> well, yes and no.
18:37:03 <nirik> yes it could easily, but most of our proxies are underpowered and I don't know that they could run openshift
18:37:15 <pingou> nice
18:37:30 <nirik> and we would need to replace all the static content they currently do. But yeah, thats an option for the coming year at least as a pilot...
18:37:49 <nirik> it would be nice to have autoscaling... "lots of mirror requests, lets fire up 5 more mirrorlist containers"
18:38:29 <nirik> Oh, here's a haproxy for proxy11... https://admin.fedoraproject.org/haproxy/proxy11
18:38:59 <nirik> 0 requests dropped. and 6ms time... all the mirrorlist servers are many times that away
18:39:35 <pingou> very nice
18:40:00 <nirik> I've not announced anything yet because I want to see if anyone notices it being faster/more reliable... but we will see. ;)
18:40:06 <nirik> any questions?
18:40:19 * relrod has to dash out and head to class
18:40:38 <pingou> have fun relrod :)
18:40:52 <nirik> safe travels.
18:40:59 <nirik> #topic Open Floor
18:41:11 * pingou grbmls at libgit2/pygit2
18:41:20 <nirik> anyone have anything for open floor? suggestions, comments, ideas, favorate breakfast foods?
18:41:31 <pingou> .bacon ?
18:41:31 <zodbot> ≋≋≋≋ BACON ≋≋≋≋
18:41:39 <wfturnage> eggs
18:42:01 <marc84> steak and eggs
18:42:18 <nirik> all good. ;)
18:42:27 * nirik is surprised pingou didn't say cheese. ;)
18:42:40 <smooge> .cheese
18:42:44 <smooge> doesn't work
18:44:28 <nirik> formage?
18:44:36 <nirik> anyhow, will close out in a minute if nothing else.
18:45:00 <smooge> I don't have anything. tomorrow I am rebuilding ibiblio04 if all goes well
18:45:52 <nirik> cool.
18:46:00 <nirik> #endmeeting