15:02:06 <cverna> #startmeeting Infrastructure (2020-06-25)
15:02:06 <cverna> #meetingname infrastructure
15:02:06 <zodbot> Meeting started Thu Jun 25 15:02:06 2020 UTC.
15:02:06 <zodbot> This meeting is logged and archived in a public location.
15:02:06 <zodbot> The chair is cverna. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:06 <zodbot> The meeting name has been set to 'infrastructure_(2020-06-25)'
15:02:06 <zodbot> The meeting name has been set to 'infrastructure'
15:02:16 <cverna> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:02:16 <cverna> #info Agenda is at: https://board.net/p/fedora-infra
15:02:16 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:02:19 <mkonecny> .hello zlopez
15:02:20 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:02:22 <cverna> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:02:22 <cverna> #topic aloha
15:02:28 * pingou ahoy!
15:02:29 <amrmzaki> .hello amrmostafazaki
15:02:30 <zodbot> amrmzaki: amrmostafazaki 'Amr Mostafa Zaki' <amr.mostafa.zaki@gmail.com>
15:02:37 <cverna> bonjour
15:02:53 <nirik> morning
15:03:04 <amrmzaki> Bonjour :D
15:03:17 <pingou> salutations1
15:03:35 <cverna> #topic New folks introductions
15:03:35 <cverna> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:03:42 <cverna> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:04:06 <cverna> Anyone new or anyone that would like to introduce themselves ?
15:04:33 <mobrien[m]> .hello mobrien
15:04:34 <zodbot> mobrien[m]: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:05:22 <tflink> morning
15:05:22 <cverna> I guess not :)
15:05:24 <cverna> #topic Next chair
15:05:24 <cverna> #info magic eight ball says:
15:05:35 <cverna> #info 2020-07-02 - mkonecny
15:05:35 <cverna> #info 2020-07-09 - nirik
15:05:57 <mkonecny> I'm guilty
15:06:06 <cverna> anyone want  to chair the 2020-07-16 meeting ?
15:06:21 * cverna sends mkonecny to jail
15:06:33 <mkonecny> No
15:06:57 <mkonecny> But I will take the chairing :-)
15:07:09 * cverna accepts some money and let mkonecny go free
15:07:40 <cverna> thanks michal
15:07:41 <cverna> #info 2020-07-16 - mkonecny
15:07:53 <cverna> #topic announcements and information
15:08:05 <cverna> #info CPE Sustaining EU-hours team has standups on Tuesday and Thursday at 1400 UTC in #fedora-meeting-2 - please join
15:08:05 <cverna> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:08:13 <cverna> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:08:20 <cverna> #info Fedora Communishift move has started but will take longer than expected. Current estimate for bringing back into production is TBD
15:08:25 <cverna> #info Blog post about MBBox published https://communityblog.fedoraproject.org/mbbox-module-building-in-a-box/
15:08:35 <cverna> #info OSBS is working again \o/
15:08:46 <nirik> hurray!
15:08:57 <mkonecny> Nice
15:09:33 <pingou> #info toddlers are up and running, loopabull has been stopped
15:09:37 <cverna> any other info ?
15:09:46 <pingou> and so far the toddlers are behaving!
15:09:52 <nirik> hurray again
15:09:58 <cverna> \o/
15:10:08 <cverna> so many good news today
15:10:12 <pingou> we have ~2500 messages queued in rabbitmq, it was 7k+ earlier today
15:10:35 <pingou> or maybe 6k+ only, still a nice drop
15:11:29 <pingou> #info we have 62 accounts in dist-git w/ no corresponding bugzilla account
15:11:50 <pingou> that is 5 less than last time and 1 more (we want from 66 to 62 in total)
15:11:53 <nirik> I may have kojira playing nicely for repo regens... still I think has issues with deletes, but thats being worked on upstream
15:12:01 <pingou> these 62 accounts have been emailed directly
15:12:15 <pingou> nirik: that's a very good news!
15:13:04 <nirik> I have bene pondering a email on side tags tho... we seem to stick around 70-100 of them... and I wonder how many people are always using them...
15:13:40 <cverna> We have the one from the monitoring script that accumulate
15:13:45 <pingou> there was/is a RFE to gather who are these side-tags from
15:13:54 <cverna> neither bodhi nor koji seems to be deleting them
15:13:55 <pingou> monitoring a bit who has the most side-tag may help
15:14:01 <nirik> 73 f33 side tags right now...
15:14:08 <pingou> from monitoring?
15:14:13 * pingou can clean them
15:14:16 <nirik> so that means when a package lands in f33, it has to do about 80 newrepos
15:14:21 <nirik> no total
15:14:27 <pingou> ok
15:14:45 <cverna> bodhi cannot delete them since admin cannot delete side tags anymore
15:14:59 <cverna> but koji should be cleaning them up I did not have time to look at it further
15:15:18 <nirik> yeah, we need more investigation there.
15:15:27 <cverna> +1
15:15:47 <cverna> moving to next topic
15:15:56 <cverna> #topic Oncall
15:15:56 <cverna> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:16:10 <cverna> #info siddharthvipul is oncall for 2020-06-25 -> 2020-07-02
15:16:27 <cverna> anyone willing to take 2020-07-02 -> 2020-07-09 ?
15:16:27 <pingou> nirik: check now?
15:16:50 <nirik> wow... 22.
15:16:59 <pingou> :)
15:17:12 <nirik> much much better
15:17:21 <nirik> cverna: I can take it.
15:17:30 <cverna> nirik:  thanks
15:17:44 <cverna> #info nirk is oncall for 2020-07-02 -> 2020-07-09
15:17:55 <cverna> #info Summary of last week: (from current oncall )
15:18:06 <cverna> mkonecny:  anything ?
15:18:35 <mkonecny> If I don't count the switch failure on Friday, it was quiet week
15:18:43 <mkonecny> Got two pings
15:18:43 <nirik> heh.
15:19:14 <mkonecny> But the Friday was not much fun
15:19:15 <cverna> switch failure does not count :P
15:19:42 <nirik> there are worse times for it to have failed... but yeah, no fun
15:19:51 <mkonecny> Ok, then it was quiet week :-D
15:20:04 <cverna> siddharthvipul: or siddharthvipul_ or siddharthvipul1 don't forget to take the oncall from mkonecny :)
15:20:25 <cverna> #topic Monitoring discussion [nirik]
15:20:26 <siddharthvipul> :p
15:20:30 <siddharthvipul> I am it, I am all
15:20:34 <siddharthvipul> .oncalltakeeu
15:20:34 <zodbot> siddharthvipul: Kneel before zod!
15:20:39 <cverna> #info https://nagios.fedoraproject.org/nagios
15:20:39 <cverna> #info Go over existing out items and fix
15:20:46 <siddharthvipul> thank you cverna
15:20:50 <pingou> nirik: https://pagure.io/fedora-infra/howtos/c/581f19d9c7b9cd4bd659cfb3fcb1ed9f17887e2d?branch=master fyi :)
15:20:52 <nirik> still no nagios. ;) so skip this week again
15:21:04 <cverna> nice and easy :)
15:21:10 <cverna> #topic Data-Center Move update
15:21:27 <nirik> pingou++
15:22:00 <pingou> nirik: do you want me to take oncall?
15:22:07 <nirik> so, all machines are now racked, most of them we can reach mgmt on... Smooge was working on the last devices... I have started on installing and re-adding builders
15:22:43 <nirik> after we get builders and qa sorted, will bring up some 02 versions of things, then... on to staging
15:23:01 <nirik> pingou: I thought siddharthvipul has it?
15:23:22 <pingou> nirik: I mean next week
15:23:45 * pingou would not want to step on siddharthvipul's toes :)
15:23:47 <nirik> ah... if you like. I should be around... also, note that next friday is a us holiday I think...
15:24:00 <cverna> thanks for the update nirik
15:24:33 <cverna> #topic Open Floor
15:24:53 <cverna> the floor is open
15:25:03 * nirik tries not to fall through
15:25:04 <siddharthvipul> pingou: if you want to take oncall, please feel free to do so. mostly I will be pinging you with doubts if I get a confusing one :P
15:25:25 <pingou> siddharthvipul: no worries, I'll have your back ;-)
15:25:47 <pingou> I have one random question: how do we want to handle tasks that should run periodically?
15:26:09 <pingou> do we want to use cron in openshift? use a "runner" like we do for monitor-gating?
15:26:23 <pingou> something generic/extensible like toddlers or on project per task
15:26:44 <nirik> well, do you have some examples?
15:26:44 <pingou> I'm thinking: we want to regularly sync packagers to bugzilla (maybe every 6h?)
15:27:05 <pingou> and we want to email people that do not have a bugzilla account (maybe every week?)
15:27:29 * nirik has a probibly too complex idea...
15:27:32 <pingou> then there is the question of removing people from retired packages (frequency to be thought about)
15:27:56 <pingou> so we have at least 3 scripts that we want to run regularly
15:28:06 <pingou> I'd be for cron if cron were not such a pain in opensfhit
15:28:15 <pingou> openshift*
15:28:31 <nirik> yeah... agreed. it could be nicer for sure.
15:29:03 <nirik> I was pondering how to do it with toddlers... but that would take us or something sending hourly or whatever messages... "it's been an hour"
15:29:14 <nirik> which is... too complicated
15:29:30 <cverna> can we do cron on a vm ?
15:29:49 <nirik> sure.
15:30:02 <nirik> I do like the idea of everything consolidated in openshift tho
15:30:19 <cverna> then cron in OpenShift :P
15:30:33 <pingou> I'll see if I can get something working a little like monitor-gating but with more scheduled items
15:30:36 <nirik> I wonder if there's another batch job operator in openshift?
15:30:47 <cverna> there are a bit painful to setup but once it is running it works great
15:30:58 <mobrien[m]> set a cron to spin up an openshift pod to run a cron
15:31:28 <nirik> ha
15:31:35 <pingou> accessing the logs is a tad painful though
15:31:47 <pingou> so debugging when something doesn't/didn't work is hard
15:32:05 <nirik> we could make them log to log01 or email a list?
15:32:11 <nirik> but of course that could break too
15:33:26 <cverna> aren't the logs aggregated in kibana for cron ?
15:33:41 * cverna never had to look at cronjob logs
15:33:41 <nirik> possibly...
15:33:55 <nirik> kibana is kinda hard to use
15:34:03 <mkonecny> I find the Kibana hard to read and copy something over
15:34:05 <nirik> or at least Id really like more training on how to use it
15:34:21 <pingou> +1
15:34:24 <cverna> hum we don't seems to have projects in kibana https://kibana.app.os.fedoraproject.org/app/kibana
15:35:09 <nirik> cverna: it's the pull down there on the left
15:35:13 <cverna> yeah the UI is bad
15:35:15 <nirik> it says operations*
15:35:22 <nirik> but you can get project*
15:35:41 <nirik> yeah, I don't care for it, but then I don't really know how to work it.
15:36:00 <mkonecny> It will be nice to get a plain text log for old pods
15:36:01 <nirik> and it might be broken perhaps
15:36:30 <nirik> there might be a way to get it to log everything to log01/rsyslog...
15:36:37 <pingou> hm actually, for distgit-bugzilla-sync which is cron-based, I do see the text logs
15:36:52 <mobrien[m]> Kibana is more of a visualisation/aggregation tool rather than for looking through raw logs I think
15:37:27 <pingou> that makes this a 4th cron-based project
15:37:43 <pingou> all of which are around bugzilla and dist-git :)
15:38:02 <nirik> this could all be an initative...
15:38:08 <nirik> 'redo logging'
15:38:25 <nirik> since epylog is dead...
15:38:29 <cverna> we have the review-stats static files build running as a cron job in openshift
15:38:29 <pingou> I'd consider redo monitoring first, but redo logging may be nice as well
15:39:23 <nirik> all stuff -> flowmon or something -> database -> greylog or something to report out
15:39:42 <pingou> Will may have some ideas/info about this
15:39:43 <nirik> anyhow, lets perhaps start a list thread?
15:39:51 <pingou> as I think he looked at this for our apache logs
15:39:52 <nirik> indeed
15:40:48 <cverna> any other topic ?
15:41:19 <pingou> nirik: do you want to chat about noggin/stg?
15:41:31 <pingou> your random idea from yesterday :)
15:41:56 <nirik> oh yeah, I was gonna send that to the list too...
15:42:00 <nirik> but I got swamped
15:42:12 <nirik> basically should we try and bring up stg with noggin instead of fas...
15:42:24 <nirik> that would let us work out integration issues...
15:42:40 <mkonecny> This is a good idea
15:42:40 <cverna> sounds like a good idea to me
15:43:21 <nirik> we will still need to sort out migration...
15:43:30 <nirik> but it could be done later after things are working in stg.
15:44:04 <pingou> we will still have to migrate prod to stg
15:44:10 <pingou> so that would be a first migration, no?
15:44:45 <nirik> no, we can do that later before we are ready to go to prod... at first we can just make a new ipa cluster and new noggin and have been working on stg make new accounts...
15:45:02 <nirik> then once everything is all working, we figure out how to migrate the fas data in...
15:45:11 <nirik> and test it in stg for a while before moving to prod
15:45:13 <pingou> ah ok, yup wfm
15:45:53 <nirik> I'll mail the list on it.
15:46:07 <mkonecny> wfm?
15:46:08 <cverna> +1
15:46:14 <mkonecny> windows file manager?
15:46:16 <cverna> works for me
15:46:19 <pingou> works for me :)
15:46:23 * pingou too slow
15:46:34 <mkonecny> Ok, new acronym to forget :-D
15:46:38 * cverna too fast too furious
15:47:26 <cverna> ok I ll close the meeting in 3min if we don't have anything else
15:50:12 <cverna> #endmeeting