14:01:29 <tflink> #startmeeting 14:01:29 <zodbot> Meeting started Mon Apr 20 14:01:29 2015 UTC. The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:01:36 <tflink> #meetingname fedora-qa-devel 14:01:36 <zodbot> The meeting name has been set to 'fedora-qa-devel' 14:01:40 <tflink> #topic role call 14:01:50 * mkrizek is here 14:01:58 * kparal 14:02:08 <tflink> #chair kparal mkrizek 14:02:08 <zodbot> Current chairs: kparal mkrizek tflink 14:02:56 <tflink> no josef today? 14:03:08 <kparal> he's on his way 14:03:32 * jskladan1 is present 14:03:42 <tflink> #chair jskladan1 14:03:43 <zodbot> Current chairs: jskladan1 kparal mkrizek tflink 14:03:51 <tflink> cool, shall we get started? 14:03:59 <tflink> who wants to go first with status? 14:04:14 <mkrizek> I will 14:04:20 <mkrizek> #topic mkrizek status report 14:04:27 <mkrizek> #info works on remote execution for disposable clients 14:04:31 <mkrizek> #info now trying to make paramiko send stdout/stderr in realtime 14:04:37 <mkrizek> #info ETA is this week 14:05:05 <mkrizek> questions? 14:05:07 <tflink> sounds good, looking forward to the review 14:05:31 <mkrizek> that's about it from me 14:06:05 <tflink> next? 14:06:36 * kparal still writing his status 14:06:46 <kparal> :) 14:06:53 <tflink> jskladan1? 14:06:58 <jskladan1> #topic jskladan status update 14:06:59 <jskladan1> #info resultsdb's slowness was caused by the absence of indexes on ForeignKeys in Postgres - should be sorted out now 14:06:59 <jskladan1> #info OpenQA runs in Docker now - we might discuss replacing Suse on the Boston machine with fedora + containers 14:08:40 <tflink> glad that we appear to have the slow query thing figured out 14:08:49 * tflink forgets - did that get deployed to dev or stg? 14:09:24 <tflink> doesn't look like it :( 14:09:42 <tflink> #action tflink to release new resultsdb and deploy to dev/stg 14:09:44 <jskladan1> tflink: not sure, to be honest - we were talking some downtime, and I'm not sure how it ended 14:10:02 <kparal> I think it did, I haven't seen those exact errors in the last few days. but I overlooked them 14:10:21 <tflink> we got rid of the long queries in dev/stg, so they wouldn't show up 14:10:24 <jskladan1> kparal: that is most probably caused by the fact, that resultsdb does not return data 14:10:26 <kparal> the problem is that the debug log is truncated, so often we can't be sure 14:10:55 <tflink> yeah, I also want to get the new libtaskotron deployed to dev/stg today to get that tested 14:11:06 <jskladan1> kparal: ready yet? :) 14:11:09 <kparal> tflink: please wait for one more patch, coming today 14:11:11 <kparal> yes 14:11:20 <kparal> #info kparal status update 14:11:22 <tflink> kparal: ok, will do 14:11:23 <kparal> damn 14:11:26 <kparal> #topic kparal status update 14:11:45 <kparal> #info the last time I played with virt-builder (disposable clients feature) was three weeks ago, in the last two weeks I was swamped with manual testing and only had time to try to debug specific errors in our buildbot logs. for many of those I submitted patches that will hopefuly fix or improve them, e.g.: 14:11:49 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D339 14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D336 14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D337 14:11:50 <kparal> https://phab.qadevel.cloud.fedoraproject.org/D323 14:11:53 <kparal> #info one more patch coming today for libtaskotron exceeding the buildot timeout when working with a lot of updates 14:12:31 <kparal> there's an avalanche of errors from buildbot, we need to do something about it. hopefully with the latest patches it will improve substantially 14:12:43 <tflink> yeah 14:12:52 <tflink> hopefully we'll have it figured out before freeze ends 14:13:08 <tflink> once the queues empty out, we hsould have fewer timeout issues either way 14:13:28 <kparal> I spent 4 hours today downloading packages to run a depcheck run and simulate some of those issues 14:13:48 <tflink> sounds like you found a new issue? 14:14:05 <kparal> it seems it is mostly caused by bodhi being slow and buildbot not detecting any new output on stdout for 40 minutes 14:14:17 <kparal> so I added some progress reports to stdout 14:14:36 <tflink> cool, sounds good 14:14:46 <kparal> originally I suspected yamlish issue returning, but all seems good 14:14:47 <tflink> do you think that the second patch will be ready today? 14:14:53 <kparal> tflink: yes 14:15:20 <kparal> I'm just stuck installing fake_fedorainfra to try posting to bodhi 14:15:32 <tflink> what's going wrong? 14:15:35 <kparal> but I'll try to make it work somehow 14:15:43 <kparal> not sure, I probably don't know how to configure it properly 14:16:37 <tflink> let me know if I can help - I thought it was documented enough to figure out reasonably but it sounds like not so much 14:16:37 <kparal> I wondered if we could deploy a public instance of fake_fedorainfra and resultsdb that anybody could use for dev purposes. and we would wipe the db regularly 14:16:47 <kparal> it would save people a lot of time :) 14:17:17 <kparal> tflink: actually, I haven't found any readme, but I haven't searched in the wiki yet 14:17:18 <tflink> or improve the docs for deploying them - I usually find that to be quicker 14:17:59 <tflink> we can look into it, though 14:18:29 <kparal> ok, we can discuss this further after the meeting 14:18:32 <kparal> that's all from me 14:18:40 <tflink> #topic tflink status update 14:18:51 <tflink> #info upgraded stg to f22, newest libtaskotron 14:18:51 <tflink> #info helped triage the timeout issue in resultsdb 14:18:51 <tflink> #info got disposable client skeleton pushed 14:18:51 <tflink> #info working on production taskotron re-deployment and prep this week 14:18:52 <tflink> #info working on more disposable client stuff (hopefully) this week 14:19:45 * tflink is planning to push a new version of libtaskotron as soon as the next patch is pushed 14:20:58 <tflink> any questions/comments? 14:21:18 <jskladan1> nope 14:21:24 <tflink> ok, moving on 14:21:33 <tflink> #topic upgrading taskotron production 14:22:00 <tflink> other than the timeout issues which are hitting dev, stg and prod - is anyone aware of any reasons not to upgrade production after freeze is over? 14:22:46 <jskladan1> nothing here 14:22:49 <jskladan1> maybe kparal? 14:22:59 <kparal> I'm all for upgrading production 14:23:01 <mkrizek> no reasons 14:23:19 <kparal> with the latest patches 14:23:22 <tflink> #info no known issues that would cause us not to upgrade production as planned after freeze is lifted 14:24:13 <tflink> freeze will be lifted tomorrow, I don't remember what time of day it's lifted but I was thinking of doing it either tomorrow, wednesday or thursday 14:24:58 <jskladan1> sounds good to me 14:25:25 <tflink> maybe wed-thurs would be better to get a bit more time to test out the new patches that will be released today 14:26:18 * tflink will send out announcements before actually doing it since this will involve downtime 14:26:48 <tflink> any other questions or comments? 14:27:02 <mkrizek> nope 14:27:11 <jskladan1> nn 14:27:16 <tflink> ok, moving on 14:27:21 <tflink> #topic beaker 14:27:38 <tflink> I found out last week that some new machines have been donated for use with beaker 14:28:02 <tflink> which kinda increases pressure to get something working as a production system instead of the current dev system 14:28:31 <tflink> spoke with the beaker devs last night and they've started working on an ansible patch to get that at least started 14:28:56 <tflink> but I suspect that they'll need help making everything work with infra 14:29:18 <tflink> the debate in my mind is how high of a priority do we make this? 14:29:39 <tflink> theoretically, the task to import new trees into beaker should be working 14:31:08 <tflink> any thoughts? 14:31:56 <kparal> taskotron infra is definitely of a higher priority, I would say 14:32:18 <kparal> I guess you're asking more about taskotron development vs beaker work 14:32:38 <tflink> yeah 14:34:15 <kparal> I think this effort should be a collaboration of several teams, our team can't do everything. so does it sound reasonable to say that we shouldn't be e.g. putting 10x times more time into it than beaker devs do? 14:34:39 <tflink> yeah, that makes sense for now 14:34:57 <tflink> hopefully we'll see the first patch in the next week or so and can move on from there 14:35:00 <kparal> it should be balanced, so I think even the times spent across different teams should be somewhat balanced 14:36:29 <tflink> in an ideal world, I'd agree :) 14:36:49 <tflink> at some point, we may need to decide how important having beaker in fedora is, but that doesn't need to be today 14:38:41 <tflink> it sounds like there aren't many more thoughts on this - mostly wanted to give a heads up and get thoughts started 14:38:56 <kparal> how much hardware do we have now? 14:39:00 <kparal> for beaker 14:39:24 <tflink> with the old boxes that are going away around october - 10 boxes, 2 aren't racked yet 14:40:02 <kparal> so 10 bare-metal clients 14:40:13 <kparal> ok 14:40:21 <tflink> bare metal and virthosts 14:40:30 <tflink> the clients that are actually in beaker are vms right now 14:40:45 <tflink> I don't think I ever got back to adding the bare metal clients after the firewall issues were fixed 14:41:26 <tflink> nope, doesn't look like it 14:42:29 <tflink> anyhow, will keep folks updated as things change 14:42:39 <tflink> it'd be nice to have a decision on beaker soon 14:42:49 <tflink> before october, I mean 14:43:23 <tflink> #topic tasks for the next week 14:43:33 <tflink> is anyone needing tasks for the next week? 14:44:18 <kparal> I don't, unfortunately 14:44:42 <kparal> mkrizek went to the bathroom, it seems 14:45:04 * jskladan1 also has enough on his plate ATM 14:45:36 <mkrizek> I have enough tasks to keep me busy for two weeks I think 14:45:48 <tflink> k, I suspect that I'll mostly be busy with infra stuff for the next several days 14:46:49 <tflink> which sounds like time for 14:46:53 <tflink> #topic open floor 14:46:59 <tflink> anything else that folks wanted to bring up? 14:48:03 * kparal nothing here 14:49:09 <mkrizek> nope 14:49:18 <tflink> ok, I'll send out minutes shortly 14:49:21 <tflink> thanks for coming, everyone 14:49:24 <kparal> thanks 14:49:26 <tflink> #endmeeting