16:30:02 <siddharthvipul> #startmeeting Infrastructure #8914
16:30:02 <zodbot> Meeting started Tue May 19 16:30:02 2020 UTC.
16:30:02 <zodbot> This meeting is logged and archived in a public location.
16:30:02 <zodbot> The chair is siddharthvipul. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:02 <zodbot> The meeting name has been set to 'infrastructure_#8914'
16:30:11 <siddharthvipul> #chair nirik mobrien[m] mboddu
16:30:11 <zodbot> Current chairs: mboddu mobrien[m] nirik siddharthvipul
16:30:32 <amrmzaki> .hello amrmostafazaki
16:30:33 <siddharthvipul> #chair astepano amrmzaki
16:30:33 <zodbot> Current chairs: amrmzaki astepano mboddu mobrien[m] nirik siddharthvipul
16:30:34 <zodbot> amrmzaki: amrmostafazaki 'Amr Mostafa Zaki' <amr.mostafa.zaki@gmail.com>
16:30:42 <mobrien[m]> .hello mobrien
16:30:43 <zodbot> mobrien[m]: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:30:51 <astepano> .hello astepano
16:30:52 <zodbot> astepano: astepano 'Andrei Stepanov' <astepano@redhat.com>
16:31:01 <nirik> give me a min... just finishing something.
16:31:02 <siddharthvipul> hello everyone :) you all must be wondering why I have invited you here
16:31:15 <siddharthvipul> nirik: sure, please take your time :)
16:31:31 * mboddu grabs something to drink in the meantime
16:32:43 <nirik> morning everyone.
16:32:50 <nirik> siddharthvipul: why did you invite us here? :)
16:32:55 <amrmzaki> hello .nirk
16:33:25 <amrmzaki> sry :) hello .nirik
16:33:41 <nirik> ok, so lets talk dns.
16:33:50 <siddharthvipul> haha, yep.. that's it ^
16:33:52 <fm-admin> pagure.issue.comment.added -- bstinson commented on ticket fedora-infrastructure#8926: "Fedora CI pipeline - slow performance" https://pagure.io/fedora-infrastructure/issue/8926#comment-653098
16:34:08 * nb is here also
16:34:26 <nirik> the way our dns is setup is using a git repo on batcave01... we make changes to that git repo and then our nameservers pull that repo to get their config.
16:34:49 <nirik> the repo is /git/dns on batcave01
16:34:51 <astepano> nirik: https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/dns.html good doc
16:35:05 <astepano> it describes all these steps
16:35:19 * nirik nods.
16:35:34 <nirik> that doc is a bit outdated now...
16:35:40 <astepano> but, sysadmin-main  is the group that can push
16:36:21 <amrmzaki> which part need to update .nirik ?
16:36:30 <nirik> and actually it's wrong in a few places. ;( would anyone like to fix it up/submit a PR?
16:36:47 <nirik> sysadmin-main and sysadmin-dns can both push changes
16:37:05 <mobrien[m]> I'll take notes and update it after
16:37:16 <siddharthvipul> mobrien[m]: thank you, if you need help, let me know :)
16:37:28 <nirik> well, not so much wrong I guess as not idea.
16:37:31 <nirik> ideal.
16:37:37 <nirik> so, in the section about making changes.
16:38:01 <nirik> we prefer now if you can to make your changes, commit those, then run the do-domains script.
16:38:13 <nirik> then commit that and push the 2 commits.
16:38:21 <nirik> that way it's much easier to see what you changed.
16:38:33 <nb> good point nirik
16:38:51 <astepano> Does `do-domains` automatically update git-index?
16:38:52 <nirik> FYI, the do-domains script signs things with dnssec and also checks for errors/issues.
16:39:19 <nirik> it doesn't. just signs and checks.. you have to 'git add -a' and 'git commit' after it.
16:39:42 <astepano> this is very important step, thank you.
16:40:11 <nirik> so, astepano you wanted a zone you all could update, right?
16:40:16 <nirik> any thoughts on the name?
16:41:07 <astepano> I do not want zone delegation. Some agreement that fedora-ci guys can add changes to fedoraproject.org.template
16:41:10 <nirik> we also have 2 new nameservers in our new datacenter... ns01.iad2 and ns02.iad2
16:41:46 <astepano> let's say under `ci` subdomain
16:41:52 <nirik> astepano: hum... can you give me an example of what you would be adding? note that messing up this zone could cause our entire infra to be down.
16:42:04 <astepano> jenkins-old.ci  IN CNAME aaa.apps.ci.centos.org.
16:42:34 <nirik> ah, for a subdomain you don't need to edit the main fedoraproject.org template...
16:42:41 <nirik> you can just edit that zone
16:43:16 <nirik> will there be any problems with ci.fedoraproject.org getting confused with any other ci stuff? :) thats my only thought... otherwise it seems fine
16:43:39 <astepano> I checked that fedoraproject.org.template
16:43:47 <astepano> no `ci` entries inside
16:44:05 <astepano> but, yeah, up to you
16:44:52 <nirik> sure, I was thinking more from it seeming like there's a number of different ci groups... testing-farm, fedora-ci... just likely my confusion.
16:45:16 <astepano> it is almost the same initiative
16:45:33 <nirik> ok.
16:45:54 <nirik> nb: would you have time to add a ci.fedoraproject.org zone? or I can do it if you don't...
16:46:29 <nirik> .fasinfo astepano
16:46:29 <zodbot> nirik: User: astepano, Name: Andrei Stepanov, email: astepano@redhat.com, Creation: 2015-03-21, IRC Nick: astepano, Timezone: Europe/Prague, Locale: en, GPG key ID: 9A82842F, Status: active
16:46:32 <zodbot> nirik: Approved Groups: aws-fedora-ci @fedora-ci-admins modularity-wg sysadmin-upstreamfirst sysadmin packager +atomic-ci fedorabugs cla_done cla_fpca
16:46:44 <nirik> ok, let me add you to sysadmin-dns
16:47:11 <nb> nirik, I can do it
16:48:03 <nirik> hum, fas is being flaky
16:48:38 <nb> nirik, worked for me
16:48:44 * nb added him since you said you were having problems
16:49:42 <nirik> ah, ok, thanks.
16:50:16 <nirik> so, siddharthvipul / mobrien[m] / amrmzaki / astepano: any questions on the sop? or how it's setup
16:50:23 <astepano> I have
16:50:34 <astepano> but, it is a bit unrelated to DNS
16:50:46 <astepano> it is about cert for HTTPS for OpenID Connect
16:51:03 <amrmzaki> .nirik that's fine
16:51:11 <nirik> astepano: so where is the service running?
16:51:24 <nb> do we need to add any of the others too? or just astepano?
16:51:40 <nirik> just astepano for now...
16:51:44 <nb> ok
16:51:44 <astepano> nirik: for example on AWS or OpenShift centos cluster.
16:52:12 <siddharthvipul> so far I have been following what's happening.. I will have to read more docs to related things to understand it exactly
16:52:18 <nirik> astepano: best to use letsencrypt... or, for aws we could use the amazon cert service... the name of which escapes me
16:52:40 <amrmzaki> :)
16:52:50 <nirik> I don't know if there's any automation about doing that with centos-ci...
16:52:50 <mobrien[m]> nirik ACM
16:53:03 <nirik> yeah, thats it. amazon has so many services...
16:53:04 <astepano> okay, and I am trying to understand how workscertgetter01: but ssh  astepano@certgetter01 doesn't allow to login
16:54:45 <nirik> certgetter works like this:
16:56:21 <nirik> in your application you have to setup proxys for the urls that letsencrypt hits (/.well-known/whatever) to certgetter01, then we have to add that to our proxies to allow you to talk to certgetter01 for that domain... then when ansible runs a playbook calls certbot on certgetter01, letsencrypt hits your site, then the proxy, then certgetter and then it apprroves the cert, then ansible moves it into place.
16:56:33 <nirik> this will not be usable for you without your setup in our ansible.
16:56:44 <nirik> so, I think best you just do it locally...
16:56:59 <astepano> ah, thank you for the explanation!
16:57:09 <mobrien[m]> I have one question about the SOP. In the `DNS update` section, should that be changed to use pagure ansible instead of directly editing the batcave ansible?
16:57:22 <nirik> if you don't want the weight of certbot, there's acme-tiny which is really simple.
16:57:38 <astepano> So, when the zone is created, do I need to modify timestamp ? Or it will be the same as for template?
16:58:03 <nirik> mobrien[m]: yep. good eye. It should use the pagure one now.
16:58:24 <mobrien[m]> ok, I'll add that to my notes
16:58:24 <amrmzaki> i think you should increase the serial id
16:58:42 <nirik> astepano: whenever you update the zone, you will need to update the serial, it should be in YYYYMMDD01 form... 01 for first change on that day, then 02 for next, etc.
16:58:59 <astepano> okay, noted. Thank you!
16:59:12 <nirik> mobrien[m]: I bet there's a number of sops wrong now... we might try and mass update them sometime.
16:59:21 <astepano> +2 the double commit rule
16:59:43 <mobrien[m]> nirik I was thinking the same thing when I asked the question
17:01:03 * nirik nods.
17:01:39 <nb> astepano, I've added ci.fedoraproject.org.  You should only need to edit master/ci.fedoraproject.org
17:01:51 <siddharthvipul> nb++
17:02:01 <astepano> nb: wow, that was quick! thank you!
17:02:05 <astepano> nb++
17:02:06 <zodbot> astepano: Karma for nb changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:02:38 <nirik> astepano: want to try an update and see if you have the process ok?
17:02:42 <nb> actually, I might need to add it to the named.conf too I think
17:02:47 <nirik> nb: yep.
17:02:58 <nb> one moment
17:03:02 * nb needs to clone the new ansible repo
17:03:11 <astepano> nirik: let's try
17:03:21 <mobrien[m]> I have one last question that is definitely related to the SOP, where is the repo for it?
17:04:07 <nirik> https://pagure.io/infra-docs/
17:04:12 <astepano> but, my `id` command still shows that I am not in the sysadmin-dns
17:04:21 <mobrien[m]> thanks nirik
17:04:22 <nirik> astepano: hang on.
17:04:39 <nb> nirik, with the new ansible repo, does pushing to the pagure repo automatically update the copy on batcave?
17:05:01 <nirik> nb: yep. it listens for commits and updates batcave on those.
17:05:07 <mboddu> nb: Yes, not to your fork though
17:05:18 <nirik> astepano: you may need to logout and back on but it should be active now
17:05:50 <astepano> nirik: works! was some delay.
17:06:03 <nirik> I forced a refresh. :)
17:07:03 <nirik> mobrien[m]: if you want another PR/change to infra docs if you have time... the ssh access SOP... we need to now mention the new datacenter. Instead of sshing via bastion.fedoraproject.org it's via bastion-iad01.fedoraproject.org...
17:07:04 <siddharthvipul> nirik: seems it's done, are we good to close the meeting and we can see if there is any doubts/questions later on?
17:07:06 <nb> nirik, can you please give sysadmin-dns rbac access to groups/dns.yml?
17:07:15 <nirik> siddharthvipul: sounds good. yep
17:07:16 <siddharthvipul> oops, please continue
17:07:28 <nirik> nb: oh, should have that, but let me look
17:07:37 <nb> it says i don't
17:07:47 <mobrien[m]> nirik sure I'll add that too
17:07:51 <nb> i can run update_dns.yml which just refreshes the zones on all the servers, but not the actual server playbook
17:07:57 <nirik> ah, I see.
17:08:36 <siddharthvipul> #endmeeting