18:00:00 <nirik> #startmeeting Infrastructure (2015-09-10)
18:00:00 <zodbot> Meeting started Thu Sep 10 18:00:00 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <nirik> #topic aloha
18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:01 <nirik> #topic New folks introductions / Apprentice feedback
18:00:16 * aikidouke here
18:00:16 * relrod waves
18:00:43 * threebean is here
18:00:43 * pingou not here
18:00:55 * lmacken 
18:00:59 <nirik> welcome everyone. ;)
18:01:06 * lbazan here
18:01:07 <mpduty> .fasname mpduty
18:01:12 <nirik> any new folks like to give a short one line introduction of themselves?
18:01:21 <nirik> or any apprentices with questions or comments?
18:02:41 <nirik> ok then. ;) moving on to info/status stuff
18:02:55 <nirik> #topic announcements and information
18:02:55 <nirik> #info We are now in F23Beta FREEZE - no changes to frozen hosts without patch/ack - everyone
18:02:55 <nirik> #info Made a list of RHEL6 hosts to help migrate them - kevin
18:02:55 <nirik> https://fedoraproject.org/wiki/Infrastructure/RHEL6_hosts
18:02:56 <nirik> #info old cloud is finally retired! - patrick
18:03:07 <nirik> anything else anyone wants to note or talk about there?
18:03:11 * SmootherFrOgZ here
18:03:24 <nirik> great timing. ;)
18:03:28 <nirik> #topic fas3 plans - kevin/smootherfrogz
18:03:46 <nirik> so, whats our plan with fas3? how close is it to being ready to deploy in stg?
18:04:22 <SmootherFrOgZ> I wrote down list of task needed to be done for a stg instance
18:04:31 <SmootherFrOgZ> on my wiki page
18:04:42 <nirik> https://fedoraproject.org/wiki/User:Laxathom/Drafts:FAS3.0
18:05:06 <SmootherFrOgZ> yup, this one
18:06:01 <threebean> nice!
18:06:06 * nirik reads it over
18:06:48 <nirik> alright.
18:07:23 <SmootherFrOgZ> there's two things left though. the 2factor auth with have already. Can we plan a short call or irc meeting to talk about. there might be a lot to tell I think
18:07:29 <nirik> so, as far as timing goes, I'd think getting to staging before f23 release and then looking at moving into prod after f23 might be doable?
18:07:45 <SmootherFrOgZ> I can simply add the feature as they are for a 1rst version
18:08:04 <SmootherFrOgZ> and then decide what we want to manage 2xfactor auth in a long term
18:08:21 <SmootherFrOgZ> nirik: yeah should be doable, yeah
18:08:26 <nirik> yeah, so now it's that if you have it enabled it's always required right?
18:08:43 <SmootherFrOgZ> I will spwan an instance to trail the new fas-client
18:09:18 <SmootherFrOgZ> nirik: yeah and a group's admin can requires it to be mandatory for their members
18:09:46 <nirik> SmootherFrOgZ: what data does the fas3-dev instance have? just random data?
18:09:53 <SmootherFrOgZ> this a way to prevent from people asking around what to do when they join a group and access hosts
18:10:00 <SmootherFrOgZ> nirik: random
18:10:09 <nirik> but it should be ok to sign up there to test with?
18:10:20 <SmootherFrOgZ> nirik: oh, yeah, we also need to start thinking of the data migration :)
18:10:29 <SmootherFrOgZ> nirik: yup, sure
18:10:49 <nirik> yeah, can we script a db migration? or ?
18:10:55 <nirik> the interface sure looks nice.
18:11:35 <SmootherFrOgZ> that's what I've in mind for right now, a scrip migration as the data model is way different
18:12:16 <SmootherFrOgZ> Do we still need to think about a FAD?
18:12:44 <nirik> not sure... if there is enough work for one yeah.
18:12:59 <nirik> hum, the "I am not a robot" thing on the last registration page seems broken
18:13:32 <SmootherFrOgZ> Oh snap
18:13:48 <nirik> no biggie. ;)
18:14:17 <SmootherFrOgZ> nirik: log with the default account, will fix that in a few
18:14:30 <nirik> so, perhaps you could send to the list the items we need to address before stg so we could discuss them wider?
18:14:49 <nirik> then if there's enough yeah, we could look at a fad in nov/dec?
18:14:50 <SmootherFrOgZ> works for me
18:16:31 <nirik> ideally we could get it in stg before too long so we have a long period of testing available.
18:17:42 <nirik> ok, any other things on it we should discuss today?
18:18:10 <aikidouke> the wiki search? can i give a readers digest of the options ive found?
18:18:24 <SmootherFrOgZ> hm... threebean I guess we will wait pingou back to schedule a sync-up?
18:18:26 <nirik> aikidouke: I meant on this topic. ;)
18:18:33 <aikidouke> oh sorry
18:18:35 <nirik> we can get to that in open floor or later in the meeting. :)
18:19:03 <nirik> SmootherFrOgZ: are most of the packages already packaged that you need? I would guess theres some overlap with bodhi2?
18:19:28 <SmootherFrOgZ> nirik: there are.
18:19:55 <SmootherFrOgZ> those who not is optional at install or will be
18:20:40 <nirik> ok.
18:20:53 <nirik> anything else on fas3 for now?
18:21:22 <SmootherFrOgZ> nope, I actually am finishing up the Admin's page fyi
18:21:46 <nirik> great, so: have a meetup next week or after, open discussion of needed stuff for stg on list.
18:21:53 <nirik> thanks SmootherFrOgZ
18:22:05 <SmootherFrOgZ> nods
18:22:05 <nirik> #topic metalink timeouts/mirrormanager issues detection - kevin
18:22:33 <nirik> so, we had a report that sometimes people are getting timeouts on getting metalinks.
18:22:37 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/4882
18:22:41 <nirik> with handy graphs.
18:23:10 <threebean> SmootherFrOgZ: agreed :)
18:23:14 <nirik> the way those requests work is
18:23:55 <nirik> user -> proxy{01-11} httpd -> haproxy -> {vpn} -> mirrorlist* (5 of them) and back the same path
18:24:07 <nirik> so somewhere in there are some random 30s timeouts.
18:24:22 <nirik> anyone have good ideas on how to isolate and fix it? :)
18:25:08 <nirik> I suspect it might be httpd at the proxies, but I don't have any data to confirm that.
18:25:24 <dotEast2015> nirik, how did get the 30s?
18:25:39 <dotEast2015> end-to-end?
18:25:46 <nirik> thats the default timeout dnf uses I think...
18:25:59 <dotEast2015> ah, I see
18:26:01 <nirik> so the end user sends request, waits 30s and no answer
18:26:13 <nirik> but it's only sporadic.
18:26:18 <nirik> and it seems to be all proxies.
18:26:47 <nirik> https://fedorahosted.org/fedora-infrastructure/attachment/ticket/4882/timeout_g.png
18:26:58 <dotEast2015> vpn might also be the issue or some where along the way
18:27:10 <nirik> I guess we could try and get several of us to re-run scripts and see if we see it...
18:27:27 <nirik> dotEast2015: it could be...
18:27:37 <dotEast2015> good to have access to web/proxy logs to track hit time
18:27:57 <nirik> yeah, but we get lots and lots and lots of metalink requests
18:28:18 <dotEast2015> I see
18:29:55 <nirik> on one proxy for yesterday we got about 1.7million requests to mirrors.fedoraproject.org
18:30:43 <nirik> I don't think there's any logging on timed out requests... on the sever anyhow
18:31:08 <dotEast2015> I might be wrong, but redirects can be traced in apache
18:31:16 <dotEast2015> through logs I mean
18:31:53 <nirik> well, it's not a redirect, it's a proxy.
18:32:06 <nirik> I do see a few... but I don't think enough to explain this:
18:32:21 <nirik> [Thu Sep 03 11:05:25.406410 2015] [proxy_http:error] [pid 29354] (20014)Internal error: [client 96.241.79.49:42583] AH01102: error reading status line from remote server localhost:10002
18:32:44 <nirik> thats an error talking to haproxy... but it could have been restarting or something.
18:33:07 <nirik> anyhow, if anyone has good ideas on how to track this down, or would be willing to run the scripts from their connection and gather more data, that would be great
18:33:31 <aikidouke> what scripts nirik?
18:33:57 <dotEast2015> I guess any that can go through the whole circuit
18:34:23 <nirik> I guess they didn't provide a full script, but the curl command in the ticket run over hours/days
18:34:36 <nirik> 10min between checks
18:35:03 <aikidouke> ok, i'll look it up
18:35:27 <nirik> running from lots of different locations might help us isolate things... but it seems to be from all the sites the reporter used. ;(
18:35:45 <nirik> ok, thanks.
18:35:54 <nirik> #topic TRAC tickets review - p_klos
18:36:02 <nirik> p_klos doesn't seem to be around...
18:36:14 <nirik> I poked some folks on some old tickets the other week...
18:36:51 <nirik> #topic search engine status
18:36:59 <nirik> aikidouke: you had some info here? ;)
18:37:06 <aikidouke> a tiny bit
18:37:46 <aikidouke> the open source options i found mostly run on java - apache solr opensearch engine are two
18:38:00 <aikidouke> apache spark claims to run either with java or python
18:38:15 <aikidouke> so alot of the options are java
18:38:25 <aikidouke> or roll-your-own
18:38:34 <nirik> yeah. ;(
18:38:50 <nirik> did you get a chance to look a dpsearch again? it was somewhat promising a while back...
18:39:53 <aikidouke> i did, and i excluded it for some reason, but cant remember why
18:40:22 <aikidouke> ah dp search is now apache spark i belive
18:40:46 <aikidouke> the latest info on dpsearch.com is 5 years old?
18:41:33 <nirik> huh.
18:42:18 <aikidouke> here is a use case of someone using spark as a search engine
18:42:19 <aikidouke> http://www.maana.io/company/press-release/the-first-and-only-big-data-search-engine-powered-by-apache-spark/
18:42:50 <nirik> slightly newer: https://github.com/Maxime2/dataparksearch
18:43:09 <aikidouke> ahh...my googlefu has failed me
18:43:34 <aikidouke> i will do some digging on that for next week
18:44:05 <nirik> yeah, might still be dead/not working, dunno. ;)
18:44:16 <nirik> anyhow, good info. ;) thanks for looking into it.
18:44:45 <aikidouke> yup
18:45:35 <nirik> #topic Open Floor
18:45:42 <nirik> anyone have any items for open floor?
18:46:06 <dotEast2015> I have one: any pieces from fas3/cloud migration for fi-apprentice?
18:46:31 <nirik> there will definitely be some fas3 work. ;)
18:46:43 <nirik> not sure on the cloud... puiterwijk has been dealing with that
18:47:03 <mpduty> fedora-easy-karma doesn't work? says cannot query Bodhi
18:47:55 <nirik> mpduty: there's some patches to get it working... I don't know the state, please check bugzilla
18:48:07 <nirik> dotEast2015: we will try and get some easyfix tickets filed. ;)
18:49:06 <nirik> if nothing else will close the meeting in a minute.
18:50:05 <nirik> thanks for coming everyone@
18:50:08 <nirik> #endmeeting