15:00:26 <siddharthvipul> #startmeeting Infrastructure (2020-09-17)
15:00:26 <zodbot> Meeting started Thu Sep 17 15:00:26 2020 UTC.
15:00:26 <zodbot> This meeting is logged and archived in a public location.
15:00:26 <zodbot> The chair is siddharthvipul. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:26 <zodbot> The meeting name has been set to 'infrastructure_(2020-09-17)'
15:00:26 <siddharthvipul> #meetingname infrastructure
15:00:26 <zodbot> The meeting name has been set to 'infrastructure'
15:00:26 <siddharthvipul> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:00:26 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:00:26 <siddharthvipul> #info Agenda is at: https://board.net/p/fedora-infra
15:00:26 <siddharthvipul> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:26 <siddharthvipul> #topic aloha
15:00:51 <nirik> morning. aloha
15:00:54 <creaked> .hello2
15:00:55 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com>
15:00:58 <siddharthvipul> .hello siddharthvipul1
15:00:59 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:01:32 <mobrien> .hello2
15:01:33 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:01:33 <pingou> ó/
15:01:41 <siddharthvipul> welcome you all :)
15:02:08 <mboddu> .hello mohanboddu
15:02:09 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
15:03:27 <siddharthvipul> #topic New folks introductions
15:03:27 <siddharthvipul> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:03:27 <siddharthvipul> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:03:35 <siddharthvipul> do we have anyone new here? :D
15:04:18 <creaked> its been awhile so I'm new here again :)
15:04:46 <siddharthvipul> creaked, welcome again ;)
15:04:51 <nirik> welcome back creaked
15:05:18 <darknao> .hello2
15:05:19 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
15:05:24 <TomAPC> .hello2
15:05:25 <zodbot> TomAPC: Sorry, but you don't exist
15:05:35 <TomAPC> ouch :(
15:05:39 <TomAPC> .hello2 TomAPC
15:05:40 <zodbot> TomAPC: Sorry, but you don't exist
15:05:48 <TomAPC> .hello TomAPC
15:05:49 <zodbot> TomAPC: Sorry, but you don't exist
15:06:45 <nirik> TomAPC: if your irc nick isn't the same as your fas account name, use .hello fasaccountname
15:06:54 <TomAPC> .hello tomapc
15:06:55 <zodbot> TomAPC: tomapc 'Tom Collins' <tcollins@teratux.com>
15:06:59 <TomAPC> Ah, case sensitive
15:07:17 <nirik> yeah, fas accounts are all lower cased.
15:07:38 <mkonecny> .hello zlopez
15:07:39 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
15:08:00 <siddharthvipul> we have a lot of people today :)
15:08:04 <siddharthvipul> let's move to agenda
15:08:12 <siddharthvipul> #topic Next chair
15:08:19 <siddharthvipul> #info 2020-09-24 - mkonecny
15:08:30 <siddharthvipul> we need a volunteer for 2020-10-01
15:08:33 <siddharthvipul> do we have anyone? :)
15:09:52 * nirik can if no one else can...
15:10:00 <mobrien> I'll do it if no volunteers
15:10:08 <siddharthvipul> I was about to say the same thing
15:10:12 <siddharthvipul> great minds think alike :)
15:10:17 <siddharthvipul> sooo.. who is it XD
15:10:25 <mobrien> fools seldom differ :D
15:10:26 <nirik> mix all three and do the next 3?
15:10:33 <siddharthvipul> mobrien, or that :P
15:10:39 <siddharthvipul> nirik, works
15:10:51 <siddharthvipul> #info 2020-10-01 - nirik
15:11:04 <mboddu> I cannot chair the meeting since sometimes I have conflicts :(
15:11:12 <siddharthvipul> #info 2020-10-08 - mobrien
15:11:37 <siddharthvipul> #info 2020-10-15 - siddharthvipul
15:11:44 <nirik> works for me.
15:11:48 <siddharthvipul> mboddu, no worries at all, we are good for a while :)
15:12:05 <siddharthvipul> #topic announcements and information
15:12:05 <siddharthvipul> #info CPE Sustaining EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 0830 UTC in #centos-meeting
15:12:05 <siddharthvipul> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:12:05 <siddharthvipul> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
15:12:07 <siddharthvipul> #info F33 Beta freeze is in effect
15:12:28 <mkonecny> #info F33 Beta is still NO GO
15:12:51 <siddharthvipul> mkonecny, thank you :)
15:13:00 <siddharthvipul> do we have anything else?
15:13:19 <nirik> #info staging is about ~80% installed
15:14:07 <mboddu> Yay, we are almost there
15:14:40 <nirik> although thats just installed. Still need to sync databases and fix all the broken things. ;)
15:15:52 <siddharthvipul> let's move ahead if we have no announcement :)
15:16:07 <siddharthvipul> #topic Oncall
15:16:07 <siddharthvipul> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:16:17 <siddharthvipul> #info pingou  is oncall for 2020-09-10 to 2020-09-17
15:16:17 <siddharthvipul> #info mobrien is oncall for 2020-09-17 to 2020-09-24
15:16:28 <siddharthvipul> before we transfer the oncall duty
15:16:42 <siddharthvipul> who will volunteer for 2020-09-24 to 2020-10-01
15:17:14 <siddharthvipul> it's a very easy and very helpful thing (if there is someone new here) :)
15:17:20 * nirik can if no one else. ;)
15:17:32 <siddharthvipul> nirik, sure thing :) thank you
15:17:46 <siddharthvipul> #info nirik is oncall for 2020-09-24 to 2020-10-01
15:17:53 <mobrien> .oncalltakeeu
15:17:53 <zodbot> mobrien: Kneel before zod!
15:18:01 <siddharthvipul> mobrien, oh you are fast :D
15:18:08 <siddharthvipul> #info Summary of last week: (from current oncall )
15:18:12 <siddharthvipul> pingou, all yours :)
15:19:19 <nirik> I think he's not around right now. :(
15:19:32 <siddharthvipul> nirik, let's get back to it later then
15:19:34 <siddharthvipul> #undo
15:19:34 <zodbot> Removing item from minutes: INFO by siddharthvipul at 15:18:08 : Summary of last week: (from current oncall )
15:19:49 <siddharthvipul> #topic Monitoring discussion [nirik]
15:19:49 <siddharthvipul> #info https://nagios.fedoraproject.org/nagios
15:20:07 <nirik> so, not much differnt that last week.
15:20:15 <nirik> some mustangs still down.
15:20:21 <nirik> some virthosts with low swap
15:20:34 <nirik> some proxies that mobrien is installing
15:21:08 <nirik> After beta is out we should make a push to clean things up...
15:21:17 <nirik> thats all, move on?
15:21:21 <siddharthvipul> awesome
15:21:42 <siddharthvipul> sooo... today the plan is to learn about fedora-infrastructure things :)
15:21:46 <siddharthvipul> and the topic for today is
15:21:55 <nirik> drumroll....
15:22:10 <siddharthvipul> #topic let's learn  openshift setup by nirik
15:22:22 <siddharthvipul> tadaaa
15:22:23 <nirik> cool.
15:22:50 <nirik> so, we have 2 openshift clusters currently. Both of them are OpenShift 3.11...
15:23:32 <nirik> mostly because we installed them back before 4.x existed, and 4.x really wants to have dedicated hardware. Our openshift 3.11 clusters are running on vm's.
15:23:49 <nirik> There's a production cluster and a staging one. Both are setup pretty much the same way.
15:24:42 <nirik> for each cluster we have a 'os-control01' vm. This is just a small vm that checks out openshift-ansible git repo, adds our inventory and templates and runs ansible to deploy the rest of the cluster.
15:25:30 <nirik> It's done this way instead of directly from our ansible host because this allows us to checkout/use a specific version of openshift-ansible and ansible without messing with our main ansible control host.
15:25:54 <nirik> That playbook deploys: 3 master hosts and 5 compute nodes.
15:26:33 <nirik> Access to the web side of the cluster is via our proxies. So user -> proxy -> openshift cluster
15:27:12 <nirik> the proxies handle the ssl side of things, so for example we can use our *.fedoraproject.org cert for 'bodhi.fedoraproject.org' which is a openshift cluster app
15:27:37 <nirik> All applications/projects are installed from our main ansible repo.
15:27:55 * pingou in another meeting indeed, I don't think I've had a single ping last week
15:27:57 <nirik> You can see under roles/openshift-apps/ and playbooks/openshift-apps/ the config
15:28:33 <siddharthvipul> was about to ask about the applications running, ^ solves that :)
15:28:35 <nirik> so a playbook runs and copies templates/config to (by default) os-master01 /etc/openshift-apps/appname/ and then uses oc to load it into the cluster.
15:29:16 <nirik> so this means you have to make any app changes via the playbooks/ansible git. Permissions are locked down to prevent changing things via web inteface.
15:29:45 <nirik> this allows us to easily redeploy the cluster and all apps... just run the cluster playbook, then each apps and everything is done.
15:30:17 <nirik> External storage is nfs and not dynamic. Ie, if we need storage we have to make a netapp volume, export it and make a volume in openshift.
15:30:28 <nirik> Database is external postgres server
15:30:28 <mobrien> ansible really is a lifesaver
15:30:36 <nirik> yeah, it's awesome. :)
15:31:00 <creaked> any plans to migrate to 4.x?
15:31:03 <nirik> So, thats the basics... any questions on the setup?
15:31:17 <siddharthvipul> that's awesome nirik
15:31:27 <creaked> what is it deployed on? stood up a openshift 4.2 cluster a few months back
15:31:45 <nirik> creaked: yes, we would like to. We have put in for some hardware for next year for a cluster. :) also... we have a few machines we may be able to install a cluster on late this year, but they will be older/slower boxes we didn't need for anything else.
15:31:48 <siddharthvipul> IMO the change in playbooks will be the biggest blocker (since everything is so nicely setup)
15:31:53 <TomAPC> thanks nirik! interesting to learn as a newbie
15:32:15 <nirik> creaked: right now, all the 3.11 ones are vm's...
15:32:27 <siddharthvipul> nirik, Fabian installed 4.5 in some vms last week, might be worth a spike.. though I guess we have enough things to deal with now :)
15:32:41 <creaked> gotcha gotcha cool stuff
15:32:50 <nirik> siddharthvipul: shouldn't need much change... the app side hasn't changed much betwee 3.11 and 4.x... it's adding operators and cluster management that are completely different
15:33:10 <nirik> yeah, I know it's possible... but it's... less supported I think?
15:33:23 <nirik> but that may be an option indeed.
15:33:48 <siddharthvipul> nirik, ah, I understand
15:34:01 <nirik> If he has notes from that I'd love to see them. :)
15:34:10 <siddharthvipul> nirik, will ask him tomorrow :)
15:34:32 <nirik> cool.
15:34:36 <siddharthvipul> anyone has any questions for nirik?
15:34:42 * nirik tries to think of anything else to mention.
15:34:42 <siddharthvipul> let's wait a minute
15:35:36 <mkonecny> The operators are making the life easier in OpenShift 4
15:35:50 <siddharthvipul> mkonecny, welll, I guess you can say that
15:35:51 <nirik> mkonecny: yeah, it's a big reason to move to 4 for sure.
15:36:16 <mkonecny> It will be nice to have deployment of some apps as operators, so everyone could easily deploy them :-)
15:36:21 <nirik> 4 also handles upgrading much nicer... you just tell it to upgrade and it does it. :)
15:36:38 <creaked> yeah one click upgrades are nice :)
15:36:54 <mobrien> Is there an eol for 3?
15:37:00 <nirik> upgrades on 3.x you have to change some variables and re-run the cluster deployment... it takes a long time and sometimes fails for anoying reasons
15:37:15 <siddharthvipul> nirik, can confirm that.. we have been upgrading it almost every week (minor version roll out pretty fast)
15:37:30 <nirik> june 2022
15:37:33 <nirik> https://access.redhat.com/support/policy/updates/openshift_noncurrent
15:37:47 <nirik> yeah.
15:38:28 <siddharthvipul> nirik, should we move ahead?
15:38:39 <siddharthvipul> thanks a lot for taking out time to do this.. we really appreciate it :D
15:38:43 <nirik> we did have a 4.x cluster (communishift) but it's down and unknown when it will be back. Hopefully someday
15:38:59 <nirik> siddharthvipul: no problem. Happy to share what I know... great to get questions!
15:40:21 <siddharthvipul> #topic summary from last week oncall
15:40:31 <siddharthvipul> #info nothing to report
15:40:39 <siddharthvipul> #topic Open Floor
15:41:18 * nirik has nothing
15:43:10 <creaked> any packaging work for this group that I could take on?
15:45:17 <siddharthvipul> creaked, are you in packagers group?
15:45:29 <creaked> no
15:45:52 <nirik> creaked: might be the folks working on the mailman3 stack could use some help... but I am not sure where they are at...
15:46:32 <nirik> https://pagure.io/fedora-infrastructure/issue/8455
15:46:45 <creaked> ok cool will take a look
15:47:35 <siddharthvipul> creaked, meanwhile, you may also want to look at reviewing packages (to understand guidelines better), packaging some new/retired and needed packages etc. These will help you get into packagers group
15:49:16 <creaked> ok thnx
15:50:35 <siddharthvipul> let's end the meeting in 2 minutes if there is nothing else
15:53:15 <siddharthvipul> ..3
15:53:18 <siddharthvipul> ..2
15:53:32 <siddharthvipul> #info thank you all for joining (or reading the logs)
15:53:36 <siddharthvipul> #endmeeting