15:03:22 #startmeeting fedora-qadevel 15:03:22 Meeting started Tue Feb 23 15:03:22 2016 UTC. The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:22 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:03:22 The meeting name has been set to 'fedora-qadevel' 15:03:22 #topic Roll Call 15:03:35 #chair kparal mkrizek lbrabec 15:03:35 Current chairs: kparal lbrabec mkrizek tflink 15:03:46 * mkrizek 15:03:52 lbrabec is a lie 15:03:57 * kparal is here 15:04:02 * jskladan is present 15:04:39 no jan, either? 15:04:56 tflink: nope 15:04:57 nope 15:04:58 just left 15:05:04 kparal: ninja'd 15:05:33 tflink: he was mumbling something running out of tissues and toilet paper... 15:06:00 that does sound like a problem ... 15:06:22 anyhow, I think we have all the folks we're going to get, so let's get started 15:06:32 #topic Announcements and Information 15:06:41 #info cli options are now forwarded to minions (D743) - kparal 15:06:42 #info stdout+stderr are now printed chronologically (D749) - kparal 15:06:42 #info Image building/grabbing is working fine on qa11 - jskladan 15:06:42 #info changes in resultsdb required for namespacing are in git - jskladan 15:06:42 #info proof-of-concept for openqa's needle organization in subdirs done - jskladan, jsedlak 15:06:43 #info dist git style checks - discussions about tasks pulling - mkrizek, tflink, kparal 15:06:45 #info various infra fixes and maintanence - mkrizek 15:06:54 any question/comments on that or additional items? 15:07:36 not here 15:07:41 nope 15:07:42 no 15:08:11 ok, moving on 15:08:17 #topic Image Building 15:08:33 jskladan: how has the building been going in dev? 15:08:45 building/distribution 15:09:05 tflink: as far as I can tell, it's working fine 15:09:25 I'd be tempted to push it up-the-chain to either stg or prod 15:09:29 whatever you feel is OK 15:09:48 that's what I was wondering about 15:10:03 let's get it into stg and if all goes well, prod by the end of the week 15:10:06 I'd go STG + make DEV download it from there 15:10:16 and push it to PROD if stuff does not explode 15:10:18 :) 15:10:31 are we going to have more building system deployments? or just one for dev/stg/prod? 15:10:43 mkrizek: just one I hope 15:10:50 where do we want the image building to happen? 15:11:00 if it's in prod, let's just wait to move it until then 15:11:16 but where do we test fixes? 15:11:54 do we need to check if the produced images are the same enough? 15:12:22 tflink: mkrizek: there is some merit to that, especially if we change the source kickstart 15:12:33 we'll probably want to run the latest images on dev/stg first and later on production 15:12:39 but it could IMHO be just tested by over-riding the source image in configuration 15:12:44 for the testing's sake 15:13:57 and then moving the KS to prod 15:13:57 since we don't really touch imagefactory's code 15:13:57 I don't feel it's necesary to have multiple image-building services 15:13:57 we could also have some other flavor 15:13:57 like "taskotron_cloud_dev" 15:13:59 that we could play around with 15:14:00 i think that there will be less use for a dev instance of the image builder long term but for the medium term, we still need to figure out building more than f23 images etc. 15:14:04 not disrupting the prod system 15:14:29 it's not just about the kickstart, any system updates can break our images 15:14:51 but that still doesn't require multiple builders in the long run, I don't think 15:15:00 I don't say we need multiple build services, but I think there should be delay between using the latest images on dev and prod 15:15:01 tflink: I agree 15:15:10 like jskladan was saying, i think that having a "dev" flavor would accomplish pretty much the same thing 15:15:23 kparal: as it makes sense, it IMHO is quite impractical, we are rebuilding once a day 15:15:24 and if stuff blows up, we can just use older images until it's fixed 15:15:37 and keeping 5 days of history 15:16:00 do we have any monitoring on the image building, especially if something fails? 15:16:29 when is the switchover supposed to happen? isn't it like 03:00 UTC? 15:16:37 tflink: nope, especially since there is no way to get the build logs (i.e. "why the hell it failed") from the failed build) 15:16:50 tflink: switchover to what? 15:17:03 when do the new images get downloaded to the client hosts? 15:17:18 jskladan: not even notificaitons of "this failed, you might want to take a look at it"? 15:17:41 at the moment, the cron-job is set to check for new builds once per hour 15:17:46 could be changed, ofc 15:18:27 I'm just wondering if it would be wise to have that set to when one of us will be around 15:18:37 tflink: nope. But listing the failed builds is easy. If there's a sane way to send email to someone, saying "this build failed", it can be done quite easily 15:19:05 jskladan: the local postfix relay should be set up on those machines 15:19:24 otherwise, routing through bastion works 15:19:27 but I did not bother with it, as the imagefactory is less than usefull for determining "what" went wrong 15:19:31 and is less likely to get stuck in queue 15:19:59 yeah but I think it'd be good to have some notice if/when things don't build successfully 15:20:06 so, technically, checking for failed builds is easy (and implemented), I'm just removing them just after the build fails now 15:20:36 how long do you think it'd take to implement email-on-failure? 15:21:30 as I said, I don't really know what to set up, and how, to be able to just email somebody@redhat.com, but if we figure this out, then the code for emailing itself is doable in an hour max 15:21:48 k, we can sync up on that outside the meeting 15:21:52 k 15:22:07 #action tflink to file ticket about base_image notifications, add required email info 15:22:16 anything else on this topic? 15:22:41 * tflink assumes not, moves on to 15:22:44 #topic Tasking and Roadmap 15:23:10 I've been getting more tickets into phabricator 15:23:26 there are more to put in and I'm waiting on results from some tickets before getting much farther, though 15:23:28 https://phab.qadevel.cloud.fedoraproject.org/tag/libtaskotron/ 15:23:42 the workboard for libtaskotron, at least is more organized 15:23:58 what's groomed? 15:23:59 the "groomed" and "on deck" columns are sorted in order of priority 15:24:11 kparal: enough information to actually work on 15:24:12 the same question for 'on deck' 15:24:37 "on deck" is the batch of tasks that should be chosen from next - the next priorities 15:24:46 ok 15:25:38 does this process make sense? are folks ok with giving it a try? 15:25:49 handy workboard, I'll try to use it more 15:25:58 makes sense to me 15:26:07 I'm still trying to figure out how to handle multiple project tickets, though 15:26:13 resultsdb isn't organized yet 15:26:21 and those tickets don't show up on the libtaskotron board 15:26:47 * tflink is planning to mess around with stg to find a decent way before making a mess and spamming everyone to death in prod :) 15:27:46 does everyone have enough to keep them busy for the rest of the week? 15:27:58 yup 15:28:13 * kparal set his notification for workboard adjustments to phab notification only, no email 15:28:25 yes 15:28:27 didn't know you could do that 15:28:30 yes 15:28:41 cool 15:28:46 that brings us to 15:28:47 https://phab.qadevel.cloud.fedoraproject.org/settings/panel/emailpreferences/ 15:28:51 #topic Open Floor 15:29:15 * tflink will be spending some time with those preferences later 15:29:22 any other topics that folks want to bring up? 15:29:23 we should probably mention today's hot topic of openqa integration 15:29:31 ah, good point 15:29:32 not sure if you've read all of that, tflink 15:29:41 all of what? 15:29:48 the discussions we had 15:29:55 where were they? 15:30:10 on irc mostly I think. tldr coming up 15:30:14 pungi4 is supposed to be deployed tomorrow 15:30:24 yeah, that's what adam said 15:30:33 openqa no longer pulls, needs to respond to fedmsgs 15:30:52 they wanted to use taskotron. but we probably can't finish the patches so soon 15:31:17 and also there's one small issue - our minions would get blocked while openqa is crunching 15:31:19 https://phab.qadevel.cloud.fedoraproject.org/T726 15:31:20 I forget what all they needed, to be honest 15:31:27 eh, I'm not so worried about that 15:31:31 we have 30 minions in prod 15:31:44 yes, for the moment, not such a problem 15:31:52 that ticket is for future consideration 15:32:00 point taken - not so worried about that for the moment 15:32:20 anyway, the end result is that they'll write a small fedmsg listener to fire off their openqa trigger 15:32:29 that should be really simple 15:32:36 what all did they need from us? 15:32:54 they originally wanted to use taskotron-trigger 15:32:56 would have been nice to have a little more specific warning 15:33:00 and report to resultsdb 15:33:06 yeah, I want to have that happen as well 15:33:29 but the only immediate concern is the triggering, and for that they can write a few lines of code as their custom fedmsg listener 15:33:32 "oh, btw, we're pushing a huge change to production tomorrow, have fun with that" 15:33:41 so in the end, we don't need to integrate it into taskotron _today_ 15:34:17 oh well, we're qa. it's normal to not be told about upcoming changes :) 15:34:23 about reporting to resultsdb, that can wait. and we'll decide whether we want to move them to taskotron, or just allow them to report to resultsdb from their openqa trigger machine 15:34:50 so that's the tldr version, just to get you into the picture 15:35:03 k, thanks 15:35:15 * tflink will read the IRC stuff after the meeting 15:35:26 and also anyone else reading this :) 15:36:01 that's it from me, no further topic 15:36:03 is there anything else on this topic? 15:36:39 no 15:36:48 any other topics to bring up? 15:37:22 * tflink takes that as a no 15:37:34 thanks for coming, everyone 15:37:39 * tflink will send out minutes shortly 15:37:44 #endmeeting