17:00:56 #startmeeting fedora_cloud_wg 17:00:56 Meeting started Wed Aug 10 17:00:56 2016 UTC. The chair is scollier. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:56 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:00:56 The meeting name has been set to 'fedora_cloud_wg' 17:01:02 #topic Roll Call 17:01:05 .hello sayanchowdhury 17:01:06 sayan: sayanchowdhury 'Sayan Chowdhury' 17:01:09 .hello scollier 17:01:11 scollier: scollier 'Scott Collier' 17:01:13 .hello trishnag 17:01:15 trishnag: trishnag 'Trishna Guha' 17:01:18 .fas rtnpro 17:01:19 rtnpro: rtnpro 'Ratnadeep Debnath' 17:01:27 .hellomynameis dustymabe 17:01:28 dustymabe: dustymabe 'Dusty Mabe' 17:01:32 .fasinfo kushal 17:01:33 kushal: User: kushal, Name: Kushal Das, email: mail@kushaldas.in, Creation: 2006-02-17, IRC Nick: kushal, Timezone: Asia/Kolkata, Locale: en, GPG key ID: 9DD5346D, Status: active 17:01:36 kushal: Approved Groups: @summer-coding magazine commops +sysadmin-darkserver sysadmin-datanommer +sysadmin-fedimg marketing @gitpym @gitpathagar cla_fedora cla_done fedorabugs packager docs cvsl10n gitliveusb-creator web gitfedora-web @gittranslation-filter sysadmin art gitfedoratv @gitgach @gitlekhonee @gitpony @git-boog @gitsparcy cla_fpca +python-sig +ambassadors gitspin-kickstarts nuancier-mentor 17:01:39 .hello jasonbrooks 17:01:42 jbrooks_: jasonbrooks 'Jason Brooks' 17:01:44 .hello maxamillion 17:01:46 maxamillion: maxamillion 'Adam Miller' 17:01:48 oops 17:01:50 .hellomynameis kushal 17:01:51 kushal: kushal 'Kushal Das' 17:02:16 #chair kushal scollier maxamillion jbrooks trishnag rtnpro 17:02:16 Current chairs: jbrooks kushal maxamillion rtnpro scollier trishnag 17:02:34 #chair kushal scollier maxamillion jbrooks trishnag rtnpro sayan 17:02:34 Current chairs: jbrooks kushal maxamillion rtnpro sayan scollier trishnag 17:02:40 no chair for me 17:02:42 :) 17:02:44 #chair kushal scollier maxamillion jbrooks trishnag rtnpro sayan dustymabe 17:02:44 Current chairs: dustymabe jbrooks kushal maxamillion rtnpro sayan scollier trishnag 17:02:46 i'll stand 17:02:49 heh. 17:02:58 it's over rated 17:02:58 welcome back from Flock for all who went. 17:03:01 what a great conference. 17:03:12 +1 17:03:13 scollier: my favorite of the year 17:03:24 it's tied with Summit for me :) 17:03:24 I finally met kushal in person! 17:03:59 :) 17:03:59 cool, so let's get started, there are no action items from the last meeting. 17:04:11 we can kick around some tickets and then open floor it. 17:04:41 #topic Ticket https://fedorahosted.org/cloud/ticket/125 17:05:07 Looks like jberkus isn't here today, moving on. 17:05:18 #topic Ticket https://fedorahosted.org/cloud/ticket/136 17:05:44 No Ian. whew. moving on. 17:05:46 .hello bowlofeggs 17:05:49 bowlofeggs: bowlofeggs 'Randy Barlow' 17:05:55 bowlofeggs: \o/ 17:05:57 #chair kushal scollier maxamillion jbrooks trishnag rtnpro sayan dustymabe bowlofeggs 17:05:57 Current chairs: bowlofeggs dustymabe jbrooks kushal maxamillion rtnpro sayan scollier trishnag 17:06:02 hi bowlofeggs 17:06:17 #topic Ticket https://fedorahosted.org/cloud/ticket/147 17:06:26 ok, i think we have jbrooks here 17:06:45 Hi, yes 17:06:59 hey scollier! 17:07:07 hiya 17:07:14 Basically, that one is pending some big rewrite of everything, but I think in the short term we can solve dusty's reported problem 17:07:17 and dustymabe, any feedback on ^ 17:07:39 jbrooks who owns that rewrite? 17:07:40 Dusty needs living links for vagrant, I showed him where he can get those 17:07:50 maxamillion, I think 17:08:08 scollier: jbrooks_. yeah I see where those links are- not great that they are in the testing location 17:08:13 but better than nothing i guess 17:08:18 dustymabe, they're the same content 17:08:24 but I know what you mean 17:08:34 indded. but we still have no guarantee those won't change either 17:08:35 the shas check out and everything 17:08:39 jbrooks_: ? 17:09:08 oh yeah 17:09:16 yeah 17:09:36 so, I'm going to try and resolve that this week actually ... the rewrite just isn't going to happen in a reasonable timeline because of other things going on 17:09:53 it's going to be a kludge, but we'll get something in place 17:09:55 In any case, this is one we can probably stop revisiting every mtg, there should be some sort of cold storage classification 17:10:26 jbrooks_ a note on the ticket would suite just fine, me thinks 17:11:16 i think we have enough on that one, let's move to next. 17:11:22 #topic Ticket https://fedorahosted.org/cloud/ticket/151 17:11:59 kushal, you touched this last, maxamillion, it has you as owner. updates? 17:12:39 I've said before that I thought this could be closed because the QA team is happy with where we last level set things unless I'm mistaken 17:12:55 yes, we can close this one. 17:12:59 maxamillion, kushal +1 17:13:29 kushal, want to do that? 17:13:40 I will go ahead and do that 17:13:47 kushal, thanks 17:13:51 #topic Ticket https://fedorahosted.org/cloud/ticket/153 17:14:06 so this one is owned by jberkus. 17:14:09 however 17:14:18 i did get a chance to chat with misc last week at flock 17:14:27 i think we are going to kick the tires on an internal instance 17:14:37 to get him familiar with deploying origin on Fedora 17:14:46 scollier: +1 17:14:49 and start some documentation on this. I'll update the ticket. 17:15:21 scollier++ 17:15:21 maxamillion: Karma for scollier changed to 1 (for the f24 release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:15:37 #topic Ticket https://fedorahosted.org/cloud/ticket/155 17:16:01 i see jbrooks_ just updated 17:16:06 i think it's good to go. 17:16:32 there are some open questions in the last comment. 17:16:42 if folks would like to chime in for jbrooks_ 17:16:47 +1 17:16:59 jbrooks_: looks good to me 17:17:08 who can do make the change? 17:17:25 s/do// 17:17:52 I can take a crack at a PR 17:17:53 I wanted to make sure that's something we as a group actually want because I've heard a lot of mixed feedback on the topic, if it is something we want then I'll do it this week 17:17:59 Ah, sweet 17:18:05 I'm going to try and use this week to clean up some of the atomic mess 17:18:14 So, I'm +1 and dusty is, who else? 17:18:15 before diving face first back into OSBS insanity 17:18:26 I'm +0, I don't care 17:19:21 Lazy consensus wins? 17:19:22 I also +0 17:19:26 Has anyone been -1? 17:19:26 jbrooks, hehe :) 17:20:17 jbrooks_ in summary, the change is to add new content so people can atomic host upgrade every two weeks? 17:20:42 scollier: we would turn off the nightly ostree builds and only do them every two weeks 17:20:43 scollier, The change is to compose a tree at the same time as the images, so the tree and images line up 17:20:57 versus the current, which is continuous, based on bodhi 17:21:16 jbrooks_ ok 17:21:27 do note that there's nothing currently in place to handle one-off releases for critical CVEs 17:21:35 Right 17:21:38 hmm. 17:21:39 I mean, we cold just fire off the script and do one 17:22:00 maxamillion: hmm - i didn't think we were going to stop doing nightly builds 17:22:00 is this to reduce some overhead for someone? 17:22:12 removing nightlys? 17:22:49 scollier: no idea 17:22:51 Colin would like it to work this way 17:23:01 dustymabe: colin wants the nightly builds stopped 17:23:06 There's some benefit w/ static deltas, making updates faster 17:23:12 dustymabe: at least that's what I understood 17:23:15 There's a desire to say, I'm on X deploy 17:23:23 ok let's just make sure we are all on the same page 17:23:23 jbrooks_ you have a note in there about a process for dealing with crit sec issues, if we can get there, I'm +1 17:23:38 there are a couple of things we could be talking about 17:23:45 there is nightly build of fedora atomic host image 17:23:55 there is compose of the ostrees 17:24:08 the nightly build is part of the compose 17:24:25 The continuous trees from bodhi would still exist 17:24:30 One could rebase to them 17:24:35 I thought what colin wanted was simply thatthere be a two week 'ref' people could follow 17:24:35 there's another updates-ostree (for lack of a better term) that happens daily as part of a bodhi process, the updates-ostree will stop 17:24:36 If that's what you wanted 17:24:46 wait, what? 17:25:09 and the images we release every two weeks are programmed to follow that ref by default 17:26:14 I think one of the problems with me implementing this is that I'm a little ignorant to the finer points of ostree management and creation (not because of lack of interest, just lack of time and the fact that OSBS derailed the last 9 months of my life) ... so I'm largely going to be learning as I go 17:26:31 maxamillion, I can help 17:26:41 I maintain the centos scripts 17:26:46 jbrooks_: +1 thanks 17:27:04 dustymabe, we all on same page now? 17:27:13 jbrooks_: how do you understand what colin was asking for? 17:27:30 scollier: I don't think so 17:27:32 dustymabe, It's all in the filed issues, and I've also talked to him 17:27:39 But we can talk to him again 17:27:55 ok - well from everything i read there is nothing that says we should stop building nightly cloud images 17:28:09 just that they should be programmed to follow the 2 week 'ref' by default 17:28:13 Yeah, did I say that? 17:28:21 no maxamillion did 17:28:47 I believe 17:29:04 My understanding is we do a compose alongside the image build, and use that by default, and bodhi continues doing its thing 17:29:22 yes, I did the ticket referenced here mentions it https://fedorahosted.org/rel-eng/ticket/6313 17:29:35 or maybe something mentioned in there does ... I can't remember now 17:29:51 doesn't matter, if that's not included in this then just ignore me 17:30:30 maxamillion: no worries 17:30:35 maxamillion, I'll follow up w/ you 17:30:41 i just still think there is value in the nightllies 17:30:49 dustymabe, i tend to agree 17:30:50 jbrooks_: thanks for following up 17:30:55 cool. 17:30:57 alright. 17:30:58 jbrooks_: feel free to pull me in to that 17:31:03 ok 17:31:07 jbrooks_: alright 17:31:17 next ticket is kickstart docs, will wait on that since jberkus is out 17:31:33 #topic Ticket https://fedorahosted.org/cloud/ticket/160 17:31:35 rtnpro, all you 17:32:10 so, we have the package for fedora-motd tested and available in F24 repo 17:33:14 I had ping'd jan to know about the workflow to include fedora-motd in F25 release, and it was said that if it's in the repo, nothing special needs to be done 17:33:37 I believe, all we need is to include the package in the cloud base image and atomic image 17:33:45 rtnpro: there is no reason we can't do it in Fedora 24 17:34:01 the sooner, the better :) 17:34:02 just would need proper testing up front and then tests added to tunirtests 17:34:24 I can come up with the test cases 17:34:27 you'll need to get some volunteers to test it and kushal / trishnag to verify your tunir tests 17:34:40 I can get folks to test it 17:34:42 most importantly we don't want to break things :) 17:34:48 I can do that :) 17:34:50 dustymabe, + 17:34:52 +1 17:34:57 trishnag, +1 17:35:05 i think that's really the big thing right now 17:35:20 sounds good to me. 17:35:22 did you rename the package ? I thought you wanted to make it more generic than fedora-motd 17:35:23 everything's always broken anyways ... why change now? 17:35:30 heh 17:35:30 Heh 17:35:35 so, how do I get the package included for the next build and testing 17:35:36 ? 17:36:01 IIRC, there was a PR/patch for the fedora cloud base image kickstart file 17:36:04 rtnpro: I think the best thing to do is to have people install it by "unlocking" the ostree 17:36:09 and installing the rpm 17:36:18 so you don't need an image build for that 17:36:28 ok 17:36:46 have them test it that way and then we can include it in the kickstart for cloud base and then the manifest for ostree 17:37:03 then we'll make sure to test the nightlies before a two week release comes out 17:37:29 I will create a test workflow for manual testing and add it to Fedora Wiki or the ticket #160, and then later translate the workflow to test cases 17:37:57 thanks rtnpro, good to go? 17:38:04 rtnpro: +1 17:38:09 yes :) 17:38:27 excellent. 17:38:31 #topic Ticket https://fedorahosted.org/cloud/ticket/167 17:38:34 kushal, ^ 17:39:05 #action rtnpro will add test workflow (and test cases) for testing fedora-motd on F24 atomic image 17:39:20 Sometimes our images fail to pass a blocking test. 17:39:39 Like in the example user log file. It can be anything. 17:39:57 The question is: should we keep the release blocking till it gets fixed? 17:40:16 also for few things (like the same example), we found it failing time to time. 17:40:24 kushal, that makes sense to me. 17:40:38 We don't know the exact cause why it comes back. 17:40:49 if it's a blocking test, something has to be fixed 17:41:07 trishnag, has replied to the list with the bug number. 17:41:12 trishnag, do you have it handy? 17:41:36 kushal: yes 17:41:42 https://bugzilla.redhat.com/show_bug.cgi?id=1353688 17:41:54 Thanks. 17:42:06 scollier, just want to hear what others think :) 17:43:05 Anyone? 17:43:19 so I don't think this particular issue is release blocking 17:43:34 we want a test there to make sure that it doesn't come back after it does get fixed 17:43:53 dustymabe, so we can move that test to a nonblocking one. 17:44:05 kushal: i think that depends on how you look at it 17:44:14 who is working on atomic host bugs in fedora 17:44:16 ? 17:44:34 no one is picking them up unless they have implications downstream somewhere 17:44:54 so if we want to wait forever to do this release then we make this bug a release blocking one 17:45:06 I think the answer to that is "nobody" 17:45:17 if we want to accept it the way it is for now until someone cares enough to spend time to fix it 17:45:26 then we make it non-blocking for now 17:45:29 I don't think anyone is specifically working on Fedora Atomic Host from a development perspective 17:45:42 yeah - there is part of the problem 17:45:57 * dustymabe included 17:46:02 * dustymabe is part of the problem 17:46:02 does anyone in this group realistically have the time to take that on? 17:46:10 dustymabe, yes (that no one is working). 17:46:34 i would love to - i just have too much other stuff going on 17:46:46 dayjob - and then i'm moving so that takes away my free tinme 17:46:58 dustymabe: exactly, I think that a similar problem with everyone else ... and it's not so much a placement of blame as it is a reality of our situations 17:47:06 dustymabe: myself included 17:47:16 maxamillion, well said. 17:47:21 right 17:47:21 same with everyone else. 17:47:27 i'm hoping things change soon 17:47:39 change and hope - two words for the gutter 17:47:55 dustymabe: hey now, those words won two presidential elections ;) 17:48:01 :) 17:48:40 anyhoo ... we should likely find a way to resolve that, either by clearing up cycles within our group or finding someone who can take that on ... I'll ask around a little and see if there's anyone who can join the effort 17:48:54 thanks maxamillion 17:49:14 maxamillion, thanks. 17:49:16 scollier: can't promise much but I'll try :) 17:49:18 kushal, looks like it may take a bit to resolve this. anything else? 17:49:20 * kushal has two items for open floor. 17:49:23 scollier, nope 17:49:24 excellent. 17:49:26 either way we should go ahead and release 17:49:35 this has been a bug for a while 17:49:37 dustymabe, agreed, on that issue. 17:49:40 dustymabe: the release went out actually :) 17:49:45 #topic Open Floor 17:49:45 really? 17:49:48 dustymabe: it just wasn't supposed to ... which was the bug 17:49:57 ah 17:49:58 dustymabe: https://getfedora.org/en/cloud/download/atomic.html 17:50:06 hmm 17:50:07 Two points. 17:50:21 when I went to the page yesterday it said that the tests had failed 17:50:29 and the latest release was over two weeks old 17:50:38 1. Trisha has started her internship in Fedora Engineering team from this Monday. She will work with our WG. 17:50:43 oops 17:50:45 trishnag++ 17:50:46 trishnag, 17:50:49 yea 17:50:55 trishnag, sorry for the typo. 17:51:00 * kushal needs rest 17:51:13 kushal: heh :) 17:51:20 2. We need to update the autocloud test cases in between any freeze on infra. 17:51:30 * trishnag is excited :) 17:51:33 No code update, but a data point for the autocloud project. 17:51:56 So, I will open a discussion in the infra list for that permission. 17:52:06 because we keep adding new tests 17:52:12 and changing things as required. 17:52:45 Any comments? 17:53:30 none here 17:53:36 dustymabe: yeah, the website is updated on a 30 minute cronjob interval so there's a period of inconsistency :/ 17:53:52 we have ~ 7 minutes left 17:53:56 oh - i thought that it really didn't update because of the failed test 17:54:10 while they are discussing that, jbrooks_ do you have a link to the openshift container builds for centos? 17:54:12 kushal: nope 17:54:38 #action kushal to write to infra for permission to update Autocloud tests while in freeze 17:54:46 scollier, I can look, I'm not sure where they are 17:54:53 * gholms has a thing for open floor 17:54:58 jbrooks_ kk. thank you 17:55:01 gholms: \o/ 17:55:14 gholms, hi there :) 17:55:20 o/ 17:55:27 scollier, there are some here https://wiki.centos.org/ContainerPipeline 17:55:29 dustymabe: it sadly released but shouldn't have ... there is definitely a bug 17:55:48 well glad we didn't block on that test this time 17:55:53 but yeah, def a bug :) 17:56:04 gholms: you going ? 17:56:40 jbrooks_ perfect. thank you 17:56:46 Shall I? Sure. 17:56:50 googog 17:57:06 I'm having trouble figuring out how to handle a cloud-init bug. 17:57:10 .bug 1344735 17:57:10 gholms: Bug 1344735 – /var/log/cloud-init.log empty - https://bugzilla.redhat.com/1344735 17:57:29 syslogd is in charge of writing cloud-init.log, but we turned that off. 17:57:44 while they are discussing that, maxamillion, any objections to me creating a ticket to containerize origin on fedora? 17:57:54 journald also stored everything that cloud-init sent to syslog, but we turned off journal persistence as well. 17:58:30 How are people supposed to debug cloud-init failures on instances when they can't log into them or turn them off? 17:58:37 gholms, I thought we have that between reboots. 17:58:37 'turned off journal persistence' 17:58:41 what does that mean? 17:58:57 scollier: none at all, that'd be great 17:59:01 That means we don't create /var/log/journal 17:59:11 (Or at least we didn't the last time I checked) 17:59:28 Oh 17:59:45 gholms: you mean the plain text version? 17:59:56 Yes 18:00:12 Do we create /var/log/journal again? We didn't used to. 18:00:18 dustymabe, we have 1 minute left here, y'all can keep the conversation going, but I need to hard stop for another meeting. 18:00:27 gholms: does this help? 18:00:28 http://dustymabe.com/2014/09/08/capture-elusive-cloud-init-debug-output-with-journalctl/ 18:00:31 dustymabe, mind ending meeting when ready? 18:00:36 gholms, I am not sure then, I will have to dig. 18:00:39 scollier: go ahead and end meeting 18:00:40 If so then inspecting a dead instance should be possible, assuming journalctl bothers to maintain some level of compatibility. 18:00:46 we'll continue in #fedora-cloud 18:00:52 Ok 18:00:53 cool 18:00:55 thanks everyone. 18:00:56 +1 18:00:57 Thank you everyone :) 18:01:02 #endmeeting