15:59:37 #startmeeting Server Working Group Weekly Meeting (2013-12-03) 15:59:37 Meeting started Tue Dec 10 15:59:37 2013 UTC. The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:37 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:59:41 #chair sgallagh mizmo nirik davidstrauss Evolution adamw simo tuanta mitr 15:59:41 Current chairs: Evolution adamw davidstrauss mitr mizmo nirik sgallagh simo tuanta 15:59:45 #topic roll call 15:59:47 morning everyone. 15:59:49 .hellomynameis duffy 15:59:49 .hellomynameis sgallagh 15:59:50 mizmo: duffy 'Máirín Duffy' 15:59:53 sgallagh: sgallagh 'Stephen Gallagher' 15:59:59 .hellomynameis kevin 16:00:00 nirik: kevin 'Kevin Fenzi' 16:00:14 Hello 16:00:58 .hellomynameis adamwill 16:01:01 adamw: adamwill 'Adam Williamson' 16:02:39 hello 16:03:30 Ok, so we have only a few more meetings scheduled before the PRD is due. 16:03:54 Shall we focus on that, or take some time to discuss the roles concepts? (Have people had time to digest any of that?) 16:04:43 * adamw read it. 16:04:56 either way. I think the current roles thread seems reasonable to me, although as goals, we may not be able to implement all those things in a first cut. 16:05:25 nirik: I don't think we'd try to do everything at once 16:05:31 .hellomynameis tuanta 16:05:32 tuanta: tuanta 'Truong Anh Tuan' 16:05:32 We'd end up slipping to 2016 16:05:35 right 16:05:40 do we agree on mitr proposal that backup/mesuring should be part of the core server ? 16:05:48 and not even separate roles ? 16:05:58 * mizmo read it but didn't read all the follow up on the list 16:06:03 I am not sure that can be done 16:06:14 #topic Backup and Monitoring 16:06:25 (I'm not sure about backup btw - should one be backing up servers, or backing up data storage only, possibly a NAS, and rebuilding servers?) 16:06:26 well maybe logging based on journald ... 16:06:27 * nirik is still mulling that over 16:06:28 it seems a bit too 'blue sky optimal case' as well 16:06:30 backup and monitoring coming as a part of that would be freaking awesome 16:06:37 Well, as far as backup, we can talk about block-level backup as a core system feature 16:06:38 backup ? how do you do that *by default* ? 16:06:39 how many people aren't going to have some kind of existing backup config to work with? 16:07:06 adamw: "people"? Most. Server deployments? Hopefully few. 16:07:10 adamw: how do you implement default backup in a new infrastructure before you bring up the infra ? 16:07:12 simo: lvm snapshots? backup before/after upgrades/role config changes 16:07:32 nirik: uhmm I did not count lvm snapshots as backus ... 16:07:44 you would have to configure backups before using them in any case. 16:07:55 nirik: Right. There's some (presently stalled) work there 16:08:07 The idea being to at least define the backup set in some standard way 16:08:18 well, yeah, 'backups' is kinda vuage there... is this for rollback? disaster recovery? auditing? 16:08:20 And then people could use their own preferred tools to actually perform that backup 16:08:25 Lvm backup is quite good. 16:08:27 a role defining the things you need to back up to back it up would be handy, i guess. 16:08:35 simo: backups obviously require some configuration at least for the destination; however snapshots/etc. might need a computer-wide infrastructure for letting each role manage its own backup. (or we just make a LVM snapshot and don't let roles affect it?) 16:08:39 (Project name is "roller-derby") 16:08:41 adamw: +1 16:09:23 adamw: +1 I'd be willing to make that one of the requirements for role approval 16:09:29 I think this is something great to add, but may be down the road, and I fear if we keep looking off at the end of the road, we will never get on the on-ramp. ;) 16:09:32 well again 16:09:34 adamw, +99 16:09:46 Ideally with API to allow you to retrieve that information from the role (since in some cases it may be dynamic) 16:09:52 sorry to break it for you, but lvm snapshots as "backups" would break freeipa based infrastructures 16:09:59 nirik: Yes; it should probably go hand in hand with actually having a "backup destination" role 16:10:03 nirik: The purpose of the PRD is to be the long-term goal 16:10:03 actually *any* cluster/replication based infrastructure 16:10:09 that's why I am a bit hesitant 16:10:14 The onramp stuff should be our next deliverables after that. 16:10:35 sgallagh: sure, but we can say: "integrated backups" without us talking about lvm snapshots or whatever for 30minutes. ;) 16:10:46 simo: That would be an argument in favor of roles having a say in the backup process, wouldn't it? 16:10:52 nirik: Yes, agreed. 16:11:10 the "right" "backup" infrastructure depends as much on the role of the server than anything else, which is why a default strategy is something I akm not sure I can agree with 16:11:55 simo: perhaps each role should include a 'backmeup' process/script, so it can dictate how it's best backed up, and that integrates with info from a backup role to run it? 16:11:55 simo: ISTM that this is something that could => should be solved by software without the user having to think about this 16:11:57 nirik: I would rather say: "proper disaster recovery strategies based on the roles installed" 16:12:03 (I might be naive) 16:12:25 nirik: i was thinking of it more as configuration data for a backup tool to consume 16:12:29 mitr: well some things can be done some not so easy 16:12:44 adamw: sure. 16:12:53 let's try, let's try to think of what value a distribution can appropriately add without stepping on toes 16:13:14 and in a way that's properly flexible 16:13:53 adamw +1 16:13:54 and lightweight...i'm not sure we want to be designing ourselves into a corner by blessing a single complete backup stack that is part of fedora server 16:13:55 backup role installed and configured, install freeipa role, backup role gets info from it and gathers up the data that freeipa role says it needs for backup and stores that where it's configured to. 16:14:21 sure, as long as the 'backup role' is just another interchangeable widget... 16:14:23 irregardless of the backup structure, one thing in common between different roles/apps would be what pieces are useful to back up and which aren't right? so the idea that the roles would ship with some list of stuff that's important to back up would be good wouldn't it? 16:14:28 sure, it gets into more details, but as a goal I think making it easy to backup roles is a good goal 16:14:39 yeh we seem to be agreed on this 16:14:47 maybe we should move on from backup and not get stuck on it? 16:14:52 +1 mizmo 16:14:53 the other consideration mitr brought up was monitoring 16:14:56 I kind of like the 'backmeup' idea. Have each role implement a backup solution itself and dump it to a tarball for a backup role to consume and stick somewhere 16:14:57 adamw: I wouldn't personally mind mandating a specific backup implementation FWIW 16:14:57 +1 mizmo 16:15:11 +1 mizmo 16:15:23 I think something similar could be the case for monitoring 16:15:43 each role includes info on what should be monitored on it? 16:15:50 +1 mizmo 16:16:01 mitr: I would 16:16:04 #info One thing in common between different roles is that they have pieces that are useful to back up and pieces which aren't useful to backup. So roles can ship with a list of stuff that's important to back up. 16:16:10 nirik: I think that's vague. Monitor how? 16:16:38 The very minimal case for monitoring is an /etc/monitor-open-ports.d where each role just puts a list of ports that should be responding to a TCP connect, and something to verify that periodically 16:16:51 the problem with both these "meta-definitions" is that we have no standard way to definie these things, are we givening ourselves the task to invent these standards ? 16:17:03 simo: adopt or invent I think 16:17:12 mitr: Listen on with what protocol? I think that's again too vague. 16:17:12 do you know of any standard ? 16:17:15 simo: We have invented RPM, we can invent someting like this 16:17:16 simo, let's not worry aobut the implementation right now. later on we can do a search to see what existing tech is out there and figure out which to adopt 16:17:17 process freeipa -> running, port 1345/tcp listening, etc 16:17:25 simo: OpenLMI, SNMP...? 16:17:28 sgallagh: what nirik said 16:17:34 let's just focus on what our users want and we can hash out details on implementation later, 16:17:39 mitr: I am not against, just one to spell it out that it may be a substantial amount of "standards" work 16:17:46 mitr: Ok that makes sense 16:17:47 we keep getting stuck in the weeds because we go too far into implementation concerns 16:18:04 True enough 16:18:11 sgallagh: The consumer side (OpenLMI/SNMP/UI?) needs defining as well, sure 16:18:16 I agree with simo... we should try and reuse or get someone else to implement wherever possible... ;) we should be integrating, but where there's nothing... 16:18:17 mizmo well if we are committing ourselves to do X I want to know that X is at least vaguely possible 16:18:20 so similarly to backup, the idea here is that the roles need to provide a list of things to monitor 16:18:30 mitr: I was responding to "do you know of any standard [for monitoring]" 16:18:35 simo, it's fair, i just dont want that turning into a 30 minute discussion :) 16:18:53 mizmo: I do not want to hash out what it means to do iut 16:19:17 mizmo: I was literally saying: spell it out in the PRD or whatever that it may require substantial standards work 16:19:31 simo: I think a minimum subset of "the following processes need to be running, the following ports should be listening" is probably a fair beginning 16:19:42 simo, i'm just coming from the POV of spending painful hours on last meeting's minutes trying to dissect it :) 16:19:45 sgallagh: you are going into the implementation now :) 16:20:06 mizmo: I think I summarized to you my position and saved you some time then :) 16:20:07 simo: Just citing a singular example so we know that there is *something* we can do 16:20:10 Without standards work 16:20:17 Anyway, I'm okay with the general answer here 16:20:32 sgallagh: nope you told what you want to do, not how to describe it, which is the thing I said it is going to be difficulyt 16:20:35 the general answer being roles ship with a list of things to monitor (ports, processes, etc.?) 16:20:41 unless you meant that that is a vague policy 16:20:55 e not actually something we write out in a role "definition file" 16:21:09 simo: Yes, as policy 16:21:22 sgallagh: nice try at a shortcut 16:21:24 so, this is making me think that roles could be packages... since they would be shipping things... 16:21:29 I think it will not cut it (see what I did ?) 16:21:30 I have some implementation ideas, but I'm not going to start on them 16:21:49 mizmo: as a starting point, yes. Having "requests per second" statistic or something else role-specific available from the monitoring system would be an eventual extension 16:22:00 nirik: I think they will resemble packages, but cannot be packages 16:22:04 nirik: at least not rpm ones 16:22:09 I don't think so 16:22:16 unless rpm adds stuff we do not have 16:22:16 * nirik wonder if we could also do 0 config monitoring 16:22:17 Let's not have that discussion right now 16:22:20 mitr, ooh so it'd give you some reasonable thresholds the thing should run under? 16:22:23 like optional dependencies 16:22:28 sorry, didn't mean to sidetrack, just thinking out loud 16:22:53 * davidstrauss lost track of time. 16:22:54 mizmo: automagical alerting in problematic situations is "deferred" I suppose, everyone wants it and everyone is handwavy about the magic required to detect anomalies 16:22:58 nirik: No problem, but that's an implementation thing that will require plenty of discussion. Later. 16:22:59 anyway I am vaguely in favor of roles defining these things 16:23:09 just do not think it is minor work 16:23:16 nirik: Yes, 0 config to get these things working, please 16:23:27 we could have our hands full with just all the definition work and 1 role on top of it as the example 16:23:39 * nirik points to http://linux-ha.org/source-doc/assimilation/html/index.html (early days tho) 16:23:54 Anyone want to phrase these things as a proposal (making it something we can start writing into a PRD)? 16:24:09 davidstrauss, we're talking through mitr's suggestions from the list to sgallagh's freeipa walkthrough, to also handle backup and monitoring for roles. we agreed that we'd like roles to provide a list of things to backup, and we're talking about roles providing some form of things to monitor (ports, process, etc.) and maybe thresholds it should operate under 16:24:44 proposal: roles should provide information on backups and monitoring to allow them to easily be backed up and monitored. Further details of this interface TBD. 16:24:57 +1 16:25:09 I wrote about this in one of my email. 16:25:11 emails* 16:25:38 nirik: +1. (this also implies that some amount of framework to support this will be present in the base server install, yes?) 16:25:54 If only to aggregate the data 16:25:58 tar? 16:26:25 davidstrauss: We're making an effort to avoid implementation discussions 16:26:26 nirik: +1, with an additional Proposal: "Fedora Server will provide a machine-wide infrastructure to use the role-provided information to perform these tasks:" 16:26:26 well, either in some base layer or all roles depend on backup/monitoring roles? I guess if they all do it could just be some base layer. 16:26:34 sgallagh: i don't think that necessarily follows 16:26:37 It's too easy to get lost in the weeds, and we need a high-level PRD soon 16:26:47 * adamw trying to think light touch and light weight 16:27:08 maybe we could have a list of general principles we're trying to follow too and list them in the prd so when people read these ideas there's a context? 16:27:13 e.g. "light touch, light weight" 16:27:24 e.g. "anti-NIH, we will use pre-existing tech when possible" 16:27:31 +1 both of those 16:27:38 mizmo: +2 16:27:40 also, "easy to disable" 16:27:52 nirik: Sorry, can you elaborate on that one? 16:27:57 it seems to me we're working towards an overall approach for what fedora server can do: distributions provide value by putting things together, and so we're talking about fedora server as a venue for providing these things called 'roles', which is essentially providing information 16:28:11 (many folks may have their own setups for backup/monitoring and may not want to use our setup, but I hope our setup is so cool they do :) 16:28:38 nirik: Perhaps "Avoid impeding existing utilities"? 16:28:44 adamw: yeah. contained sets of packages, configuration and information... 16:28:52 sgallagh: sure, that works too 16:29:05 the information that 'if you install X and Y and Z and configure them in this particular way you will have a web server which you can back up by preserving the files A, B and C and monitor by looking at ports G, H and I' 16:29:05 nirik: Many folks have a unique way of storing the data for safekeeping, but the needs for creating a good archive of the data are less diverse. 16:29:36 adamw: "you can monitor by looking at ports" shouldn't be something to concern the user about: they should just get the results of the monitoring IMHO 16:29:47 mitr: i think you're trying to dumb down too far. 16:29:53 okay so i'll info those principles? 16:29:58 the thing that keeps popping into my mind is YaST 16:30:08 !YaST 16:30:10 #info High-level operating principals: 1) "Light tough, lightweight". 2) "Anti-NIH: Use pre-existing tech when possible." 3) "Avoid impeding existing utilities" 16:30:18 i do not want YAST-for-servers, where i have to use precisely the configuration and tooling you want me to use or it doesn't work 16:30:26 adamw: +many 16:30:27 e.g. People may store their MariaDB backups in S3, on RAID, on Rackspace Cloud Files, or on something else, but they all use mysqldump or xtrabackup. 16:30:29 adamw: of course the format the roles will specify this information will be public and usable by other tools, but we should actually provide _an_ end-to-end solution out of the box 16:31:13 #undo 16:31:13 Removing item from minutes: 16:31:19 #info High-level operating principles: 1) "Light tough, lightweight". 2) "Anti-NIH: Use pre-existing tech when possible." 3) "Avoid impeding existing utilities" 16:31:25 adamw: nobody wants it, or in other words: on my dead body 16:31:46 adamw: See principal 3) 16:31:57 davidstrauss: sure, but some place might have their own setup to run those backups, perhaps from a central host, or storing somewhere specific or whatever. If our role says: "backup this by running mysqldump --foobar" thats fine, but the user should be able to do their own thing. 16:31:58 We should absolutely not be doing anything that precludes using other tools to make changes 16:32:07 grr.. principle 3 16:32:21 * sgallagh shakes fist at five years working with Kerberos 16:32:26 okay so we were talking about monitoring 16:32:31 i think we agreed about backups and monitoring 16:32:39 and then came up with some principles to guide the PRD 16:32:43 what do we talk about next 16:32:51 nirik: Sure. But we could package it as, say, a systemd timer unit + service that you can easily enable/disable. Just one way we could ship it without impeding. 16:32:59 mizmo: is there any list of items we need for PRD? 16:33:07 also, do we want to choose some shortlist of initial roles? 16:33:19 well i think the other two groups have drafts already so we could crib off of those 16:33:25 nirik: I think we can avoid the shortlist for the PRD 16:33:27 davidstrauss: sure. Just so long as it's easy to disable and do your own thing. ;) 16:33:29 the workstation one has personas so we should use our personas for ours i think 16:33:41 As long as we set down some intents for the roles, we can select them later 16:33:41 we should have our principles in it too 16:33:58 I see the cloud one doesn't include a cloud server. Are we making that and they are concentrating on images? 16:34:01 do we put the UX post with the free ipa example sgallagh wrote up in the prd? 16:34:05 sgallagh: ok 16:34:14 nirik: I've been trying to get an answer out of them on that. 16:34:16 #topic what goes in the PRD 16:34:49 I like sgallagh's high-level principles. 16:35:02 * sgallagh would like to quote you on that ;-) 16:35:40 Yeah, those three principles should probably be the driving part of our Overview section, I would think 16:36:26 personas should be there too 16:36:40 One thing we've talked about circuitously in the past that maybe should be a principle... 16:36:40 Thinking about my own use, I'd want something that "just works" on my personal server and is a good example of best practice (that we wouldn't use directly) for my corporate cases. 16:37:02 "Fedora Server core and roles should not require a running X server" 16:37:10 sgallagh: +50 16:37:16 third party applications may, but nothing we ship should mandate it 16:37:21 (IMHO) 16:37:28 sgallagh: Implementation detail IMHO 16:37:29 sgallagh: And any GUI tools should be remote-usable. 16:37:31 too specifically stated, given that we're all going to wayland, but the principle, sure 16:37:56 sgallagh: What about the supposed future of user-space virtual terminals? 16:38:17 mitr: I'll admit to being ignorant on the subject. 16:38:31 mitr: Do you mean graphical or non? 16:38:57 mitr: There's quite a bit going on in systemd multiseat w.r.t. graphical terminals. 16:39:08 davidstrauss: I mean that the existing "text mode" VT in the kernel might be going away in favor of a graphics mechanism (Wayland? framebuffer? I don't know) + an user-space terminal emulator 16:39:26 IMHO, I would love to have: configuration of roles can be via a gui, a tui, or a unattended mode. 16:39:34 Perhaps rephrased as "Neither the Fedora Server nor the promoted roles may strictly require the use of a local graphical environment"? 16:40:14 nirik: I'd go so far as to suggest that that unattended mode should be a MUST, but I'm flexible on the UIs 16:40:28 yeah, something like that. 16:40:29 sgallagh: I can't see the point really. Does this really mean "There shall be a CLI", or "there is a tradition of disabling the GUI 'for security'", or something else? 16:41:01 mitr: there really shall be a CLI 16:41:05 mitr: Mostly it's "The GUI pulls in too much stuff both in terms of attack surface and overall disk usage" 16:41:05 sgallagh: Agree on the unattended mode, but that doesn't preclude e.g. drawing Cockpit-like graphis on screen when nobody is logged in. 16:41:16 I agree with the sentiment, but not sure this needs to be a principal. 16:41:24 ok 16:41:31 nirik, we have that in the role install vision statement i sent to the list after last week's meeting - we came up with that text about word for word 16:41:39 mitr: it's a problem if it requires to use resources it shouldn't 16:41:41 so the role install vision statement should probably go in the PRD 16:41:46 mizmo: ok. yeah. 16:42:23 #info configuration of roles can be via a gui, a tui, or a unattended mode. 16:42:39 #info Fedora Server core and roles should not require a running X server 16:42:56 simo: To me that is a technical tradeoff, not a product philosophy matter 16:43:04 mizmo: My rephrase is probably better there. 16:43:05 mizmo: a graphical server 16:43:13 #undo 16:43:13 Removing item from minutes: 16:43:13 not X 16:43:21 #info Fedora Server core and roles should not require a running graphical server 16:43:23 local graphical server? 16:43:33 anyhow, EBIKESHED 16:43:57 ok, so let's move on 16:44:20 what else do we need for prd? 16:44:21 #topic Personas 16:44:34 Just to confirm, no one has lingering issues with the Persona definitions? 16:44:48 If so, let's give them the go-ahead for inclusion in the PRD. 16:45:00 Have they been altered much in the past week? 16:45:20 where are they? 16:45:26 think this is something i missed. 16:45:44 lemme grab the link 16:45:44 davidstrauss: You are the last editor 16:45:47 https://fedoraproject.org/wiki/Server/Personas 16:45:51 https://fedoraproject.org/wiki/Server/Personas 16:45:52 #link http://fedoraproject.org/wiki/Server/Personas 16:45:55 some need more work but i really need help 16:45:58 mitr: given remotization of displays is not a solid technology, something that depends on the graphical stack is always seen as a slippery slope to having tools that work only if you are at the local consol 16:46:01 console 16:46:02 davidstrauss did a great job with the dceisoin maker one 16:46:15 mizmo: we can change the requirement to be: all tools must be usable re motely over ssh 16:46:21 I would be fine with that definition 16:46:35 I keep thinking there might be some more we are missing, but then I draw a blank on what they might be. ;) 16:46:35 simo: s/over ssh// and I'm +1 16:46:44 mitr: =1 16:46:47 -1 16:46:47 err +1 16:46:50 must be over ssh 16:46:56 even if it is vnv over ssh 16:46:59 err vnc 16:47:00 since the last time we discussed personas, i added an additoinal developer one, so we have traditional developer and devops developer. i also added a server roles developer, and the decision maker which davidstrauss filled out 16:47:01 or spice 16:47:03 or whatever 16:47:12 simo: How about "securely" and leave that vague 16:47:29 sgallagh: well I would like to have a single port that Must be open 16:47:40 ohh, this kinda thing. ehh. 16:47:41 * adamw honestly hates this kind of 'invented user' thing and can't get a handle on writing or discussing them at all, so probably won't have much useful input... 16:47:41 simo: I think we're bikeshedding again 16:47:48 21064 for example will rarely be open 16:47:55 mizmo: devops developer might just be 'devops' 16:47:55 you *wil* have to tunnel via ssh anyway 16:48:10 adamw, we already spent a lot of time justifying the personas before you joined the group :) 16:48:11 sgallagh: which means if you rely on certs it may break them 16:48:19 nirik, fair enough i can change that now 16:48:23 simo: "single open port" with mlutplexing behind it is IMHO mostly marketing to appease firewall people 16:48:24 mizmo: i'll take it as read, then... 16:48:29 because the browser will see it as localhost:fwdwd-port 16:48:30 adamw: its just a way to see the areas that might be affected by some decision... but we will see how useful they end up 16:48:44 mitr: it's a matter of life unfortunately 16:48:51 mizmo: Who do you need help from? I'd suggest that maybe we should book a meeting with just those people during this week to nail it down 16:48:54 nirik, oh haha it does say devops not devops developer 16:49:00 ok, cool. 16:49:20 I'm fine with the personas as we have them for now, but we can always modify with more feedback from others. 16:49:36 sgallagh, i don't know who would be best able to help - what would be ideal is folks in each role sanity checking the one that best pertains to them 16:49:50 does anybody here identify with any of the roles? 16:50:13 mizmo: The decision-maker is basically an autobiography 16:50:15 mizmo: I'll sanity check 3) and 6) 16:50:23 mizmo: of me 16:50:35 sgallagh, okay thanks! 16:50:48 davidstrauss, lol yep :) 16:50:48 #action sgallagh to sanity-check "Traditional App Developer" and "Server Role Creator" personas 16:51:20 I can try on #1 (used to be there) 16:52:20 #action nirik to sanity-check "SysAdmin MacGuyver" 16:52:40 #action davidstrauss to sanity-check "Decision-Maker" 16:53:17 sgallagh: I just did, adding some edits about referrals. 16:53:24 so we need a junior sysadmin 16:53:25 hehe 16:53:58 nirik: Think you might ask one of your junior FI admins to look at that? 16:54:01 I can ask our infrastructure list... we have a number of apprentices. 16:54:02 Maybe puiterwijk? 16:54:03 yeah. 16:54:05 ha. 16:54:35 I have someone at my office who can answer what it's like to be a *senior* sysadmin at a huge company. 16:54:44 #action nirik to ask for help from the infra list for "Junior Enterprise SysAdmin" 16:55:01 mizmo: does 3 need more meat ? 16:55:13 simo, it does :( 16:55:14 rbergeron: Can I talk you into reviewing http://fedoraproject.org/wiki/Server/Personas#Persona_.232:_DevOps for us? 16:56:07 sgallagh: are you going to take 3 and 6 ? 16:56:17 do you mean just review or beefing up 3 too ? 16:56:35 simo: If you want to take one of them off my shoulders, I'm good with that too 16:56:55 And yes, in my mind, review means fixing it where it's broken. 16:57:44 sgallagh: I guess I could try to take 3 16:57:52 Thanks 16:58:02 thanks guys 16:58:04 #action Simo to review "Traditional App Developer" 16:58:23 Did we miss any other than DevOps? 16:58:30 sgallagh: sure 16:58:33 (That one I'm less worried about, since we hashed it out in an earlier meeting) 16:58:39 rbergeron: Thanks! 16:58:56 #action rbergeron to review "DevOps" 16:59:27 Ok, we're at the top of the hour. One more question on these reviews: 16:59:44 Can we set a deadline of Friday to have them complete? Is that feasible for everyone? 17:00:31 will try 17:00:41 will also try 17:00:55 okay i have a hard stop now 17:01:01 #info We will try to have the Persona reviews complete by Friday. 17:01:07 +1 17:01:36 I have a stop as well, so let's take any further topics to the mailing list or #fedora-server for now. 17:01:40 Thanks folks. 17:01:48 thanks sgallagh, everyone. 17:01:53 thanks :) 17:01:58 #endmeeting