14:00:01 <smooge> #startmeeting Infrastructure (2018-05-31)
14:00:01 <zodbot> Meeting started Thu May 31 14:00:01 2018 UTC.
14:00:01 <zodbot> This meeting is logged and archived in a public location.
14:00:01 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:00:01 <zodbot> The meeting name has been set to 'infrastructure_(2018-05-31)'
14:00:01 <smooge> #meetingname infrastructure
14:00:01 <smooge> #topic aloha
14:00:01 <smooge> #chair smooge relrod nirik pingou puiterwijk tflink
14:00:01 <zodbot> The meeting name has been set to 'infrastructure'
14:00:01 <zodbot> Current chairs: nirik pingou puiterwijk relrod smooge tflink
14:00:05 <smooge> Good morning all
14:00:09 <abompard> Hello!
14:00:10 <bowlofeggs> .hello2
14:00:12 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
14:00:12 <contyk> o/
14:00:13 <abompard> .hello2
14:00:13 <relrod> morning
14:00:15 <zodbot> abompard: abompard 'Aurelien Bompard' <aurelien@bompard.org>
14:00:16 <creaked> .hello2
14:00:17 <contyk> .hello psabata
14:00:17 <vgolwala> Hello, good morning!
14:00:17 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com>
14:00:20 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
14:00:27 <creaked> hello everyone
14:00:37 <puiterwijk> Hi
14:00:37 <vgolwala> .hello2
14:00:38 <zodbot> vgolwala: vgolwala 'Vismay Golwala' <vgolwala@redhat.com>
14:01:05 <linuxmodder> .hello2 linuxmodder
14:01:06 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@gmail.com>
14:01:14 <msuchy> hi
14:01:28 <LuckyDudeThakur> .hello2
14:01:29 <zodbot> LuckyDudeThakur: Sorry, but you don't exist
14:01:42 <smooge> #topic New folks introductions
14:01:43 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
14:02:50 <FrostyX> Hello guys, I am new here and I want to introduce myself. My name is Jakub Kadlcik (fas: frostyx) and I am a Red Hat employee working on Copr. I want to become Fedora Infra Apprentice in order to be fully capable of maintaining Copr machines.
14:02:51 <contyk> this is the first time for me, so -- it came up during the modularity wg meeting last week that someone from that group should attend these
14:02:53 <contyk> so here I am
14:02:56 <nirik> morning
14:03:08 <bowlofeggs> welcome contyk!
14:03:17 <bowlofeggs> welcome FrostyX!
14:03:23 <bowlofeggs> oh, hey nirik
14:03:28 <msuchy> I will watch over FrostyX and will learn him about playbooks and so on
14:03:29 <nirik> welcome both. :)
14:03:37 <smooge> hello all
14:03:42 <FrostyX> Thank you
14:03:47 <contyk> it's exciting!
14:03:56 <vgolwala> Hello everyone, I am Vismay Golwala. I am a Red Hat Intern currently working with the Fedora Infra team.
14:04:05 <bowlofeggs> welcome vgolwala!
14:04:15 <abompard> how are you bowlofeggs?
14:04:38 <bowlofeggs> abompard: well, i've got an americano in one hand and IRC in the other so not bad
14:04:44 <nirik> welcome vgolwala. Yoou are going to be working on bodhi this summer?
14:04:47 <wadadli> .hello
14:04:47 <zodbot> wadadli: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
14:05:06 <vgolwala> nirik: Yes I am
14:05:40 <smooge> #topic announcements and information
14:05:40 <smooge> #info relrod PTO 9 Jun - 19 Jun
14:05:40 <smooge> #info smooge PTO 8 Jun, 15 Jun -> 18 Jun
14:05:40 <smooge> #info bodhi-3.8.0 deployed
14:06:05 <smooge> Any other announcements. I am not sure about the bodhi being correct
14:06:19 <linuxmodder> nfm
14:06:28 <nirik> #info staging openshift reinstalled with 3.9
14:06:28 <bowlofeggs> smooge: it is correct
14:06:40 <bowlofeggs> sorry, i forgot to update the gobby myself, but it's right :)
14:06:42 <nirik> (today will be prod... wheee)
14:06:49 <wadadli> nirik ⟿   oh!
14:07:33 <smooge> good luck nirik
14:07:37 <puiterwijk> nirik: it'll be fine surely
14:07:44 * smooge goes on PTO
14:07:49 <puiterwijk> nirik: oh, did I inform you I'll be out from 20:00 UTC?
14:07:50 <nirik> thanks. I think with what we fixed in stg we should be fine.
14:08:13 <nirik> 🙂
14:08:20 * linuxmodder is hoping to get much more active again
14:08:44 <smooge> ok next up
14:08:48 <smooge> #topic Oncall
14:08:48 <smooge> #info Smooge is on call from 2018-05-29->2018-06-04
14:08:48 <smooge> #info Nirik is on call from 2018-06-05->2018-06-11
14:08:48 <smooge> #info Smooge is on call from 2018-06-11->2018-06-15
14:08:48 <smooge> #info (!Smooge|!Relrod) is on call from 2018-06-15->2018-06-18
14:09:42 <smooge> I decided to put a schedule up for the next couple of weeks so we have an idea of who can do things when
14:09:58 <puiterwijk> smooge: cool. thanks!
14:10:06 <nirik> smooge: good plan with all the pto coming up
14:10:36 <smooge> the last day is up in the air and needs to be filled out. we can fill out the rest of the month next week
14:12:42 <smooge> in general being on call is looking to be something for people in sysadmin main because it takes running playbooks, acking nagios, and such. I still owe everyone a sop on it
14:12:46 <smooge> #topic Tickets discussion
14:12:46 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2
14:13:14 <smooge> ah man.. I didn't take the items off the list from last meeting
14:13:32 <smooge> I thought i was so ahead of the game with gobby too :)
14:13:33 <nirik> oh right. any chance to move those forward ?
14:13:57 <bowlofeggs> i wonder if a non-main person could still be helpful as an oncall
14:14:00 <nirik> I will mark some more for meeting next week... I didn't get to it this week
14:14:12 <bowlofeggs> because they could still know whether something is important enough to interrupt
14:14:14 <bowlofeggs> and can triage
14:14:35 <nirik> bowlofeggs: I think so if they knew enough to route things / could handle easy things.
14:15:00 <bowlofeggs> i could be like the person you get when you call your isp
14:15:11 <smooge> ....
14:15:11 <wadadli> LOL.
14:15:16 <bowlofeggs> "did you try turning it off an on again?" "ok, try clicking the start menu..."
14:15:19 <puiterwijk> bowlofeggs: some ISPs you actually get a technical person as soon as you call? :)
14:15:23 <bowlofeggs> hahaha
14:15:24 <bowlofeggs> not mine
14:15:44 <bowlofeggs> sorry for derailing the topic with my senseless humor
14:15:47 <puiterwijk> But yeah. "Oh, you're not running Windows? Well, then we don't support you"
14:15:53 <puiterwijk> bowlofeggs:nah, it's fine
14:16:09 <smooge> "press  ℵ if you want to talk to a real person"
14:16:14 <bowlofeggs> #info fedora infra only supports windows users
14:16:30 <puiterwijk> #undo
14:16:30 <zodbot> Removing item from minutes: INFO by bowlofeggs at 14:16:14 : fedora infra only supports windows users
14:16:34 <smooge> ok
14:16:39 * puiterwijk does not want that in searchable logs.... :)
14:16:45 <puiterwijk> (not out of context at least)
14:16:59 <relrod> hah
14:17:10 <smooge> yeah.. someone might make us stick to that
14:17:34 <contyk> long ago they actually asked me "what operating system are you running?"
14:17:46 <contyk> "gentoo"
14:17:52 <contyk> "I've never heard of Windows Gentoo"
14:18:10 <contyk> it was a good day
14:18:11 <nirik> ha
14:18:19 <smooge> back to the meeting. I don't have any progress on the 2 tickets I took. I will remove them from the meeting queue though
14:18:20 <wadadli> heh.. no ticketts?
14:18:34 <nirik> anyhow, we can try oncall with someone else like bowlofeggs as his and our time permits?
14:18:54 <bowlofeggs> i'm up for it, perhaps in july though
14:18:56 <nirik> perhaps just try a day or something.
14:19:01 <nirik> and see how it goes
14:19:07 <bowlofeggs> june is a crazy month for me
14:19:32 <nirik> sure.
14:19:34 <smooge> #topic Migrating from fedmsg to AMQP - jcline and abompard
14:19:47 <puiterwijk> On the migration: do it, make it so. Do it yesterday
14:19:57 * nirik is fine with the plan.
14:20:16 <jcline> So a week ago, we posted to the mailing list: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/HLOYNCI4X6ELF76HP54UBMMNL4FPDBQW/ - there's not been much feedback so I just wanted to bring it up here in case people wanted a live discussion
14:20:22 <nirik> We might try set a deadline/make sure we get everything done in some reasonable timeframe tho.
14:20:37 * nirik doesn't like being in a indeterminate state.
14:20:47 <smooge> hey can we let jcline get his part out first
14:21:11 <smooge> jcline, abompard you have the floor
14:21:19 <jcline> That's pretty much all I had. I would like developers to read over https://fedora-messaging.readthedocs.io/en/latest/ and let me know if they like or don't like the APIs
14:21:45 <jcline> Now's the easiest time to change these things, so early feedback is good. I'm going to take silence as complete satisfaction :)
14:21:54 <bowlofeggs> haha
14:22:00 <nirik> oh, when the plan says 'deploy a broker' could we deploy 2 or 3? or does that not work/would add too much complexity?
14:22:09 <jcline> It should be super easy to set up and play with locally, and if it's not it should be so let me know
14:22:18 <bowlofeggs> i think rabbit can cluster, right?
14:22:28 <bowlofeggs> (for HA?)
14:22:29 <jcline> nirik, you could (and I'd recommend) deploying a cluster of 3 or 5
14:22:35 <abompard> yeah it depends if we cluster over LAN or WAN
14:22:41 <jcline> Clusters need to be on a LAN
14:22:44 <nirik> great. I thought so, but wanted to make sure.
14:22:53 <bowlofeggs> rabbit is greta
14:23:05 <jcline> You can federate between datacenters with shovel or similar plugin which uses an AMQP client to send the messages
14:23:06 <nirik> 3 probibly would be ok
14:23:40 <bowlofeggs> i have a slightly related tangent on this to discuss when we're ready
14:23:41 <jcline> I set up a cluster of 3 at home and it was easy-peasy
14:23:48 <bowlofeggs> gg ez
14:24:04 <nirik> I just want to avoid SPOF and also be able to reboot things for updates or whatever
14:24:11 <jcline> So an action item if everyone is pleased with this would be to set up the cluster in stage, then we can deploy the bridges and iron out problems
14:24:16 <bowlofeggs> erlang apps can live upgrade which is insane
14:24:29 <smooge> what is the CPU/memory/disk space we want for the systems
14:24:58 <bowlofeggs> jcline's testing was all on raspberry pis so i think we just need 3 of those on wifi
14:25:06 <jcline> For our current message throughput, anything at all basically
14:25:19 <bowlofeggs> yeah we really send very few messages
14:25:47 <nirik> yeah, they likely could be pretty small.
14:25:53 <jcline> With a lot of queues more cores help, and memory is always good.
14:26:23 <abompard> jcline: will we have a lot of queues? Probably not, right, like one per machine?
14:26:27 <jcline> But yeah, I queued up ~600k small messages on a 3 cluster including a pi3 and it was okay
14:26:34 <puiterwijk> I think jcline's test cluster vastly outperformed what we need... And it was on a RPi...
14:26:38 <puiterwijk> Yeah, that
14:26:50 <smooge> ok sounds good. We can use the arm boxes
14:26:54 <jcline> Stats on my tests are at https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/performance.html
14:27:27 <jcline> abompard, I don't think we'll have many queues, and especially at the beginning we'll have 1 or 2 for the bridges
14:28:05 <smooge> next up would be the questions nirik had about a deadline
14:28:08 <jcline> I'd expect each app to have at least one queue, but not many more than that
14:28:17 <abompard> agreed
14:28:53 <jcline> Hmm. Well, the tools are ~alpha quality, but they are tiny so writing tests and whatnot shouldn't take long
14:29:11 <jcline> It'd be great if we could have the broker running in prod by flock, even if nothing is using it yet
14:29:42 * nirik nods
14:29:42 <jcline> And I'd say we should aim to migrate everything off fedmsg by the end of the year? It should be pretty trivial to migrate each app, it's just there's a lot of them to track down
14:30:09 <abompard> and we need to decide on the security/auth/authz policy
14:30:20 <smooge> ok so the staging instances need to be ready by mid-June. staging to prod by mid-july. showoff in beginning of august. turn off in November
14:30:26 <jcline> We'd keep the external zeromq socket running so anything outside our infra won't break, so we just need to worry about our apps
14:30:37 <jcline> smooge, haha, perfect
14:30:56 <jcline> abompard, yes, we do
14:31:01 * jcline looks at puiterwijk
14:31:02 <bowlofeggs> let's move the infra meeting from irc to AMQP
14:31:05 <wadadli> who is the consumer of the message queue?
14:31:13 <wadadli> other apps or people?
14:31:25 <jcline> wadadli, other apps
14:31:27 <abompard> apps are people.
14:31:29 <abompard> sorry
14:31:33 <jcline> Haha
14:31:48 <wadadli> jcline ⟿   ok and do they do things based on the messages they receive?
14:31:56 <puiterwijk> jcline: yeah. I'll meet up with you or abompard sometime soon to talk about that
14:31:56 <abompard> hopefully
14:32:06 <jcline> wadadli, correct
14:32:14 <wadadli> ok so this is like network IPC
14:32:19 <wadadli> gotcha
14:32:23 <jcline> exactly
14:32:45 <jcline> puiterwijk, cool, should I schedule a meeting of some variety? video call or text?
14:33:15 <puiterwijk> jcline: let's try a video call sometime so I get an idea
14:33:47 <jcline> puiterwijk, okay, I'll see about putting something on the calendar next week. Sound good, abompard?
14:33:54 <puiterwijk> Yep, sound good to me
14:34:01 <abompard> perfect
14:34:07 <smooge> ok anything more on this?
14:34:07 <jcline> (that'll give me time to set up rabbit 3.7 and play with the topic auth)
14:34:17 <jcline> That's all I've got
14:34:25 <abompard> yep
14:34:27 <bowlofeggs> i have a tangent
14:34:36 <smooge> that is acute
14:34:37 <abompard> (I mean: that's all I've got too)
14:34:57 <smooge> bowlofeggs, you have the floor
14:35:25 <bowlofeggs> tangent: i'd like to be able to use this same rabbit deployment for bodhi's task workers
14:35:31 <puiterwijk> bowlofeggs: +1
14:35:43 <jcline> It'd be good for FMN to also use it
14:35:45 <bowlofeggs> mostly i'm thinking "hey, there is a nice rabbit server, let's use it for more than just fedmsgs"
14:35:49 <abompard> indeed
14:35:50 <puiterwijk> bowlofeggs: I would like to use the same cluster for every rabbit use we do. BUT first the authn/authz stuff
14:36:01 <nirik> as long as there are no security concerns, sure.
14:36:09 <bowlofeggs> right now bodhi uses fedmsg to send tasks to backend02, and i'd like to use celery or something similar instead, wtih rabbit
14:36:09 <jcline> rabbit has virtual hosts, so we should set up a virtual host for the general pub-sub stuff, one for bodhi, one for fmn, etc
14:36:35 <puiterwijk> jcline: right. But aren't clusters inside a virtual host?
14:36:39 <bowlofeggs> cool, that's my only thought :)
14:36:46 <puiterwijk> i.e. are virthosts within clusters or clusters within virthosts?
14:36:50 <bowlofeggs> i'd long wanted to do this, but i didn't want ot stand up a rabbit just for bodhi
14:37:11 <jcline> puiterwijk, I think it's many virtual hosts within a cluster, and a cluster is made up of 1 or more rabbits, but I'll need to re-read the docs
14:37:15 <puiterwijk> bowlofeggs: agreed. Right now we have a basic rabbitmq role that makes it simple, but if we get a centralized broker, that'd be great
14:37:26 <puiterwijk> jcline: ah, cool.
14:37:41 <puiterwijk> Well, if we can figure the authz stuff out, I'd like to put all the rabbit stuff on the same cluster
14:37:55 <jcline> Sounds good to me
14:38:08 <puiterwijk> Less work for everyone.
14:38:11 <abompard> yeah
14:38:19 <jcline> less work++
14:38:20 <nirik> RMQAAS
14:38:42 <puiterwijk> nirik: ......https://wiki.openstack.org/wiki/Cue
14:38:47 <puiterwijk> Don't ask. It's openstack
14:39:00 <nirik> awww... they didn't call it 'Watership Down' ?
14:39:19 <bowlofeggs> that's all from me... for now!
14:39:59 <smooge> #topic Apprentice Open office minutes
14:39:59 <smooge> #info A time where apprentices may ask for help or look at problems.
14:40:17 <smooge> any questions on tickets or such?
14:40:59 <nirik> I'd like to note our easyfix ones:
14:41:02 <nirik> https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix
14:41:29 <nirik> if anyone wants to hear more details about any of those, just ask
14:43:19 * wadadli takes a look
14:44:09 * FrostyX saves the page so he takes some issue from it in the future
14:45:01 <nirik> we also have packaging work:
14:45:06 <nirik> .tiny https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&classification=Fedora&email1=infra-sig&emailassigned_to1=1&emailcc1=1&emailtype1=substring&list_id=8012091&order=bug_id&product=Fedora&query_based_on=&query_format=advanced
14:45:06 <zodbot> nirik: http://tinyurl.com/y8pn93p6
14:46:03 <nirik> The fun never ends. ;) 🤡
14:46:27 <creaked> willing to help with packaging; jwould just need some guidance to get started
14:47:23 <smooge> ok any questions on them?
14:48:03 <smooge> #topic Open Floor
14:48:04 <creaked> smooge: i'll reach out after the meeting if you will be around
14:48:39 <smooge> creaked, I should be but slow to respond.. today is meeting day
14:49:08 <wadadli> can you guys weigh in on this? I have machine -> vm -> nested vm -> docker containers
14:49:20 <linuxmodder> wadadli,  usecase ?
14:49:27 <wadadli> I tried adding port forwarding on vm to reach docker containers from machine
14:49:44 <linuxmodder> academic or you attempting to create a insane nesting / embedded environment
14:50:06 <wadadli> linuxmodder ⟿   just ran minishift on a vm and was left with this
14:50:19 <linuxmodder> minishift ?
14:50:20 <wadadli> would like to reach web console from machine
14:50:23 <nirik> in theory I would expect you could get that to work, but why the nested vm?
14:50:38 <linuxmodder> and that^
14:50:56 <wadadli> nirik ⟿   well I have a fedora-server vm that I do most of my testing on
14:50:58 <linuxmodder> docker with at most a selinux_context-{foo}_t
14:51:28 <wadadli> so I ran the minishift installer on it and it created a nested vm
14:51:37 <nirik> ah... right. ok
14:52:10 <wadadli> but the weird thing is even though I added the forward ports in firewalld I cannot access the web console from the machine
14:52:18 <nirik> you should be able to port forward I would think... make sure there's no firewalls on the vm
14:52:32 <nirik> (in the way)
14:52:33 <smooge> vm1 or nested vm
14:53:14 <wadadli> I tried firewall-cmd --permanent --add-forward-port=port=8443:proto=tcp:toport=8443:toaddr=192.168.42.50
14:53:37 <wadadli> perhaps there might be a firewall in the way on the boot2docker instance
14:53:37 <nirik> well, the console is just on 443 I thought?
14:54:05 <nirik> anyhow, can discuss this out of meeting?
14:54:09 <wadadli> oh well after running minishift it says the console is on 192.168.42.x:8443
14:54:50 <nirik> ok.
14:54:54 <smooge> ok time to call it quits here.
14:55:05 * nirik hasn't used minishiift much... perhaps #minishift could sort it?
14:55:07 <smooge> thank you all again for coming
14:55:11 <smooge> #endmeeting