16:00:50 <mkonecny> #startmeeting Infrastructure (2022-02-17)
16:00:50 <zodbot> Meeting started Thu Feb 17 16:00:50 2022 UTC.
16:00:50 <zodbot> This meeting is logged and archived in a public location.
16:00:50 <zodbot> The chair is mkonecny. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:50 <zodbot> The meeting name has been set to 'infrastructure_(2022-02-17)'
16:00:50 <mkonecny> #meetingname infrastructure
16:00:50 <mkonecny> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:50 <zodbot> The meeting name has been set to 'infrastructure'
16:00:50 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mkonecny mobrien nirik pingou siddharthvipul zlopez
16:00:57 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
16:00:57 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:57 <mkonecny> #topic greetings!
16:01:11 <phsmoura> .hi
16:01:12 <zodbot> phsmoura: Something blew up, please try again
16:01:14 <mkonecny> .hello zlopez
16:01:14 <zodbot> phsmoura: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:17 <zodbot> mkonecny: Something blew up, please try again
16:01:20 <zodbot> mkonecny: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:25 <petebuffon> .hello petebuffon
16:01:26 <zodbot> petebuffon: Something blew up, please try again
16:01:29 <zodbot> petebuffon: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:42 <mkonecny> Welcome everyone, it seems that zodbot is blowing up :-D
16:02:14 * bcotton is here, but won't bother zodbot
16:02:27 <nirik> morning
16:03:23 <nirik> zodbot: reload Fedora
16:03:23 <zodbot> nirik: Kneel before zod!
16:03:23 <mkonecny> The zodbot works fine, just the .hello is blowing up
16:03:23 <mkonecny> I hope everyone is excited about today's meeting
16:03:31 <mkonecny> We even have learning topic today
16:03:35 <darknao> .hi
16:03:36 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:03:56 <mkonecny> .hello zlopez
16:03:57 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:04:18 <mkonecny> With the zodbot back we can start with the first topic
16:04:21 <nirik> it seems to get in a state like that after a while...
16:04:23 <mkonecny> #topic New folks introductions
16:04:23 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:23 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:45 <mkonecny> Is anybody new here?
16:05:54 <mkonecny> Feel free to speak up :-)
16:05:57 <mkonecny> We don't bite
16:06:07 <SudhanAruna97> Hi, I'm new to this group. I was GSOC mentee in 2020. Wanted to get back contributing some project.
16:06:32 <mkonecny> Welcome SudhanAruna97
16:06:50 <mkonecny> Do you have anything specific you want to work on?
16:07:07 <SudhanAruna97> currently no idea..
16:07:22 <eddiejennings> Welcome :D
16:07:24 <SudhanAruna97> maybe I'll start with easyfix issues..
16:07:34 <mkonecny> Good choice
16:07:43 <nirik> welcome SudhanAruna97.
16:08:13 <mobrien> Welcom SudhanAruna97
16:08:14 <mobrien> s/Welcom/Welcome/
16:08:20 <darknao> welcome SudhanAruna97 !
16:08:41 <SudhanAruna97> Hi everyone..
16:08:58 <petebuffon> welcome!
16:10:30 <mkonecny> Let's continue
16:10:34 <mkonecny> #topic Next chair
16:10:34 <mkonecny> #info magic eight ball says:
16:10:40 <mkonecny> #info chair 2022-02-17 - mkonecny
16:10:41 <mkonecny> #info chair 2022-02-24 -petebuffon
16:10:41 <mkonecny> #info chair 2022-03-03 -???
16:11:04 <mkonecny> Who wants to host this meeting in the March?
16:11:55 <mkonecny> Any volunteer?
16:13:28 <eddiejennings> I'll be fresh off of honeymoon, but I can do 3/3
16:13:58 <mkonecny> Sold to eddiejennings
16:14:17 <mkonecny> #info chair 2022-03-03 -eddiejennings
16:14:44 <mkonecny> What else do we have in agenda?
16:14:49 <mkonecny> #topic announcements and information
16:14:55 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:14:55 <mkonecny> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:15:02 <mkonecny> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:15:02 <mkonecny> #info matrix sig is forming, see https://discussion.fedoraproject.org/t/bi-weekly-meeting-for-matrix-sig-or-matrix-team/35947/3 if interested
16:15:03 <mobrien> congrats on getting married  eddiejennings
16:15:09 <mkonecny> #info Bugzilla will introduce breaking Authentication API change on 28th February
16:15:41 <nirik> #info Beta freeze starts next tuesday ( 2022-02-22)
16:15:43 * mkonecny joining gratulations
16:16:00 <nirik> #info tomorrow is a Red Hat recharge day, many 'hatters will be away.
16:16:38 <mkonecny> Any other announcement?
16:16:49 <nirik> oh, two more:
16:17:06 <nirik> #info mass update/reboot completed yesterday, please watch for problems with apps caused by updates.
16:17:22 <nirik> #info 2 staging virthosts are still down from updates, working to recover them.
16:18:39 <mkonecny> It seems that there is plenty that is happening
16:19:06 * nirik nods. busy busy
16:19:28 <mkonecny> Let's continue to Friend on the phone topic
16:19:34 <mkonecny> #topic Oncall
16:19:34 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:19:34 <mkonecny> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:19:34 <mkonecny> ## .oncalltakeeu .oncalltakeus
16:19:42 <mkonecny> #info dtometzki on call from 2022-02-11 to 2022-02-17
16:19:42 <mkonecny> #info mkonecny on call from 2022-02-18 to 2022-02-24
16:19:42 <mkonecny> #info ??? on call from 2022-02-25 to 2022-03-03
16:19:56 <mkonecny> .oncalltakeeu
16:19:56 <zodbot> mkonecny: Kneel before zod!
16:20:20 <mkonecny> Thanks zodbot
16:20:59 <mkonecny> So we have free plane for the friend on the phone for the 2022-02-25 to 2022-03-03
16:21:11 <mkonecny> s/plane/place/
16:21:29 <mkonecny> Who want to take it?
16:21:43 <petebuffon> i can take
16:21:48 <eddiejennings> Alas, I'll be offine until 2/28
16:22:28 <mkonecny> Who wants to take it then?
16:23:46 <nirik> I guess petebuffon said he could? :)
16:24:02 <petebuffon> indeed
16:24:10 <mkonecny> Ok, so it's petebuffon  then
16:24:29 <mkonecny> #info petebuffon on call from 2022-02-25 to 2022-03-03
16:24:46 <mkonecny> Let's continue
16:24:47 <mkonecny> #info Summary of last week: (from current oncall )
16:25:04 <mkonecny> dtometzki: Any pings?
16:26:34 <mkonecny> It seems he is not on the meeting :-/
16:26:58 <mkonecny> Let's move
16:26:59 <mkonecny> #topic Monitoring discussion [nirik]
16:26:59 <mkonecny> #info https://nagios.fedoraproject.org/nagios
16:26:59 <mkonecny> #info Go over existing out items and fix
16:27:19 <nirik> so nagios looks kinda sad right now due to those two down stg virthosts. ;(
16:27:51 <nirik> we also have some mgmt interfaces that are up, but not reachable outside their net. Hopefully we will get those fixed soon.
16:28:30 <nirik> otherwise the normal stuff we have been carrying for a while.
16:28:31 <mobrien> I see I lied last week, I need to update those certs
16:28:43 <mobrien> I said I'd have it done by now
16:28:47 <nirik> the apps.ocp one? yeah.
16:28:56 <mobrien> ya
16:29:02 <nirik> and it might be safe for me to re-upgrade the fedoraproject wildcard one.
16:29:16 <mobrien> I have a call now but I'll do it after
16:29:36 <nirik> there's some mgmt interfaces with:
16:29:41 <nirik> HTTP WARNING: HTTP/1.1 400 Bad Request - 909 bytes in 0.127 second response time
16:30:06 <nirik> which is caused by it assuming the domain part of it's hostname
16:30:30 <nirik> so it thinks it's foo.iad2.fedoraproject.org, but it's really foo.mgmt.iad2.fedoraproject.org
16:30:36 <nirik> I will go fix those soon
16:30:59 <nirik> There's also a copr check we should ask the copr folks about...
16:31:08 <nirik> "Build 3525012 took 392s (allowed 360) "
16:32:08 <nirik> anyhow, thats all I had moniting wise.
16:32:26 <mkonecny> Thanks nirik
16:32:38 <mkonecny> And now Ladies and Gentlemans
16:32:47 <mkonecny> The moment you are here for
16:32:54 <mkonecny> #topic Learning topic
16:32:54 <mkonecny> #info 2022-02-17 - Docs pipeline [darknao]
16:33:02 <darknao> ah!
16:33:09 <mkonecny> darknao: The floor is yours
16:33:22 <darknao> Ok so i'll try to describe this as best I can
16:33:36 <darknao> I gathered most of the following information by looking around, so I may have overlooked a few aspects of it.
16:33:47 <darknao> nirik, please feel free to chime in anytime and correct me if needed :)
16:34:07 <nirik> :)
16:34:07 <darknao> Alright, let's start with a quick introduction to the doc environment
16:34:18 <darknao> Fedora Documentation uses Antora as a framework for its build pipeline
16:34:24 <eddiejennings> yum, docs! :)
16:34:24 <darknao> But what Antora is exactly ?
16:34:31 <darknao> Antora is a multi-repo website generator for AsciiDoc documentation (https://antora.org/)
16:34:42 <darknao> To keep it simple, what Antora does is:
16:34:52 <darknao> - fetch documentation content from multiple git repositories
16:34:59 <darknao> - convert AsciiDoc files to HTML
16:35:10 <darknao> - then wraps it into a static website using Handlebars html templates (https://handlebarsjs.com/)
16:35:23 <darknao> There is a lot more than Antora can do, but I'll not cover it here
16:35:33 <darknao> You can check the documentation if you want to learn more about it (https://docs.antora.org/antora/latest/)
16:36:02 <darknao> The build configuration is defined via an antora playbook
16:36:14 <darknao> This is a simple yaml file listing all content repositories and configuration
16:36:28 <darknao> Our playbook is located in the main documentation repository: https://pagure.io/fedora-docs/docs-fp-o/blob/prod/f/site.yml
16:36:44 <darknao> All Fedora related documentation repositories can be found in this playbook
16:36:58 <darknao> Most of them are on Pagure, but we also have a few on Github and Gitlab
16:37:28 <darknao> The html templates, which control the look and feel of the resulting website is packaged in a "UI bundle"
16:37:46 <darknao> That's an archive that contains Handlebars templates and all the static files we need like css, js and other assets
16:37:56 <darknao> You can find the sources of the bundle we use here: https://pagure.io/fedora-docs/fedora-docs-ui
16:38:24 <darknao> Any questions so far?
16:39:23 <darknao> Ok, moving on to the main topic: the build pipeline
16:39:40 <darknao> The actual build happens on Openshift
16:39:56 <darknao> The ansible playbook for that part is located here: https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/docsbuilding.yml
16:40:13 <darknao> We have 2 jobs, one for the english site which uses the previously mentioned playbook, and another for the translated versions
16:40:27 <darknao> The 'en' version is rather quick to build (~ 5 min) and runs every hour
16:40:43 <darknao> Translated versions are much longer since we are building pretty much the same thing for each language (around 40 at this time)
16:41:12 <darknao> This last one only runs once a day and I think takes a little more than 2 hours to complete
16:41:49 <darknao> Building the translated versions of the documentation is a little bit more complex than just running the antora playbook, and requires a few extra steps.
16:42:05 <darknao> First, we need to extract translatable strings from the original content
16:42:14 <darknao> provide them to the translators via Weblate (https://translate.fedoraproject.org/)
16:42:29 <darknao> then create translated adoc files which will be used by antora in the final build process
16:42:43 <darknao> These translated document files are here: https://pagure.io/fedora-docs/translated-sources
16:43:06 <darknao> Note that, unfortunately, not all pages are translated, so we'll need to combine translated and untranslated pages at build time
16:43:20 <darknao> The job responsible of all that part also runs on Openshift, once a day
16:43:33 <darknao> The ansible playbook is https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/docstranslation.yml
16:43:42 <darknao> and the scripts behind that are here: https://pagure.io/fedora-docs/translations-scripts
16:44:06 <darknao> After the build is complete, the generated website is stored on a shared volume mounted on both sundries01 (and 02) in /srv/docs
16:44:32 <darknao> We also have old Fedora docs (F26 and below) available on this same server in /srv/web/docs-old that come from the this repo https://pagure.io/fedora-docs-web
16:44:55 <darknao> Content located on sundries are not yet accessible to the user, and that is where proxies comes in
16:45:38 <darknao> We have a bunch of proxies around the world (which also host other Fedora websites) that fetch content from sundries via rsync, combine both old & new docs into a final location in /srv/web/docs-combined, and serve them to users
16:45:55 <darknao> That is where https://docs.fedoraproject.org/ points to
16:46:16 <darknao> And... that's pretty much all I have :)
16:46:26 <darknao> Any questions?
16:47:02 <mkonecny> Is something reused for the translated builds?
16:47:05 <bcotton> o/
16:47:08 <nirik> I'd love to make this quicker/smoother...
16:47:37 <nirik> I wonder how hard it would be to leverage openshift more and have it only do builds when commits happen instead of cron job...
16:48:12 <mkonecny> Is there any plan to switch between Fedora versions for documentation? Like docs.fedoraproject.org/F35
16:48:12 <darknao> mkonecny: unfortunately, translated builds are independant from 'en' site, which means there is a lot of duplicated data
16:48:13 <bcotton> nirik: the difficulty there aiui is that there are dozens of repos to monitor
16:48:18 <nirik> and if there's a better way to sync to the proxies, or perhaps we should just ditch the proxies and serve direct from openshift (although that might make things slower for some people)
16:48:45 <nirik> bcotton: yeah. Not sure how well it would handle that
16:49:02 <darknao> mkonecny: but we are using hardlinks pretty much everywhere to limit the space used
16:49:13 <nirik> also, dropping the old docs (or moving them off to someplace) would be a very big win for infra. :) They take up sooooo much room
16:49:42 <darknao> yeah, about space usage, here are the numbers:
16:49:50 <bcotton> darknao: is there a way for mere mortals to look at build logs? let's say I think I may have broken the build...could i check on my own or would I have to ask someone in infra to look?
16:49:57 <darknao> Each site takes about 400Mo. Times 40 for all translated versions, we end up with ~16Go of documentation
16:50:09 <darknao> Add to that the old docs, which represent ~4Go, which makes a total of 20Go more or less
16:50:20 <darknao> And with hardlinks, we can lower this total to ~11Go
16:50:57 <mkonecny> Still a big amount of data for documentation
16:51:07 <darknao> bcotton: logs are only available on Openshift, and require specific access to view them
16:52:21 <darknao> the easier way right now to check if you have broken something, is to try building it locally from the docs.fp-o repo
16:52:27 <petebuffon> wow that's a lot of docs
16:53:29 <darknao> yeah there is a lot of room for improvement in this build process
16:53:31 <nirik> there's all the old fedora versions...
16:53:48 <nirik> (which was done with publican, which made epub, pdf, etc)
16:55:26 <darknao> mkonecny: I'm not sure to understand what you mean by switching between Fedora versions? we already have that right?
16:56:21 <mkonecny> We do?
16:56:30 <mkonecny> How can you do this then?
16:56:41 * bcotton misses Publican
16:57:31 <darknao> well, we have https://docs.fedoraproject.org/en-US/fedora/f35/ and https://docs.fedoraproject.org/en-US/fedora/f34 for instance, is that not what you mean?
16:57:41 <nirik> I didn't even know you were shooting at it! :)
16:58:00 <bcotton> .fire nirik
16:58:00 <zodbot> adamw fires nirik
16:58:19 <mkonecny> darknao:  Yeah, this is it, my bad
16:59:11 <mkonecny> I just always go to Engineering Teams on the page and completely ignore the top of the page :-D
16:59:14 * nirik has another meeting, have to head out. Thanks darknao and mkonecny!
16:59:57 <mkonecny> We are at the end of you time
17:00:18 <mkonecny> Thank @darknao for the learning topic :-)
17:00:27 <mkonecny> #endmeeting