16:31:45 <bgilbert> #startmeeting fedora_coreos_meeting
16:31:45 <zodbot> Meeting started Wed Mar 25 16:31:45 2020 UTC.
16:31:45 <zodbot> This meeting is logged and archived in a public location.
16:31:45 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:45 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:45 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:47 <zodbot> lorbus: In #fedora-meeting-1 is Fedora Infrastructure (starting in a day)
16:31:50 <zodbot> lorbus: In #fedora-meeting-1 is Fedora Packaging Committee (starting in a day)
16:31:51 <bgilbert> #topic roll call
16:31:55 <lorbus> .hello2
16:31:55 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:31:58 <bgilbert> .hello2
16:31:59 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:32:01 <darkmuggle> .hello2
16:32:02 <zodbot> darkmuggle: darkmuggle 'None' <me@darkmuggle.org>
16:32:10 <lorbus> .hello2
16:32:12 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:32:16 <jlebon> .hello2
16:32:17 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:38 <dustymabe> .hello2
16:32:39 <davdunc> .hello2
16:32:39 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:45 <bgilbert> #chair lorbus darkmuggle jlebon dustymabe davdunc
16:32:45 <zodbot> Current chairs: bgilbert darkmuggle davdunc dustymabe jlebon lorbus
16:32:52 <cyberpear> .hello2
16:32:53 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:33:00 <jbrooks> .hello2
16:33:01 <zodbot> jbrooks: Sorry, but you don't exist
16:33:03 <skunkerk> .hello sohank2602
16:33:08 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:33:21 <bgilbert> #chair cyberpear jbrooks skunkerk
16:33:21 <zodbot> Current chairs: bgilbert cyberpear darkmuggle davdunc dustymabe jbrooks jlebon lorbus skunkerk
16:33:57 <rfairley> .hello2
16:33:58 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com>
16:34:03 <shivarammysore> .hello shivarammysore
16:34:03 <zodbot> shivarammysore: Sorry, but you don't exist
16:34:15 <bgilbert> #chair rfairley shivarammysore
16:34:15 <zodbot> Current chairs: bgilbert cyberpear darkmuggle davdunc dustymabe jbrooks jlebon lorbus rfairley shivarammysore skunkerk
16:34:36 <jlebon> rfairley: nice to see you! :)
16:34:42 * bgilbert waves at rfairley
16:34:46 <red_beard> .hello redbeard
16:34:47 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:34:54 <bgilbert> #chair red_beard
16:34:54 <zodbot> Current chairs: bgilbert cyberpear darkmuggle davdunc dustymabe jbrooks jlebon lorbus red_beard rfairley shivarammysore skunkerk
16:34:55 <dustymabe> wow - quite the crowd here today
16:35:02 <dustymabe> welcome all!
16:35:10 <rfairley> jlebon: you as well :)
16:35:18 * rfairley waves back at bgilbert
16:35:43 <lorbus> o/
16:35:43 <rfairley> sorry it's been a while :)
16:35:58 <dustymabe> #topic Action items from last meeting
16:36:24 <dustymabe> #info no action items from last meeting \o/
16:36:48 <dustymabe> #topic RFC: Adding a Network TUI
16:36:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/427
16:37:19 <dustymabe> I'll give some background on this
16:38:03 <dustymabe> basically downstream in RHCOS land we've had some users who are having a lot of trouble with providing static networking configuration via kernel command line arguments (when required for fetching ignition configs)
16:38:39 <dustymabe> we'd like to provide some sort of interactive way for people who are doing bare metal installs to specify the network setup on their machine
16:39:20 <dustymabe> The interactive element typically goes against what we recommend for *CoreOS stle machines that are provisioned via Ignition
16:39:30 <dustymabe> *style
16:40:20 <dustymabe> but we may be able to provide something interactive without compromising our values here
16:40:36 * dustymabe waits for discussion before bringing up possible solutions
16:40:53 <dustymabe> is there anything I missed in the framing ?
16:41:34 * jbrooks installed a bare metal cluster a while back and this was painful, so it sounds interesting
16:43:29 <jlebon> dustymabe: i think you can carry on :)
16:43:54 <dustymabe> jlebon: +1 - was writing up the next stuff in a separate window :)
16:44:06 <dustymabe> possible solutions to the problem:
16:44:08 <dustymabe> 1: detect on firstboot (ignition boot) that networking isn't set up and present a TUI to the user
16:44:10 <dustymabe> 2: use the live ISO installer environment to allow users to inspect the hardware profile and provide a TUI to allow them to interactively configure networking that can then be passed to the installed system
16:45:18 <dustymabe> will pause here again for discussion
16:45:29 <dustymabe> before adding comments about the approaches
16:45:32 <red_beard> dustymabe: is there a reason why some of that couldn't be: configure on vanilla fedora and then snapshot state of some configurations to ignition/kernel command line?
16:45:34 <jlebon> is this TUI specific to *COS in any way? or could we imagine it being part of NM itself?
16:46:08 <dustymabe> jlebon: one proposal for the TUI to use is nmtui, so yes
16:46:09 <bgilbert> jlebon: I'm imagining the network part as exactly nmtui, maybe with a small wrapper menu
16:46:15 <red_beard> that way it separates concerns allows folks to noodle around as needed and then convert a working NM configuration into the needed setup
16:46:40 <red_beard> (sorry, missing commas in there breaking up thoughts)
16:46:44 <dustymabe> red_beard: isn't that pretty much what 2 is? except we're just using a FCOS interactive live environment
16:46:51 <bgilbert> red_beard: we've been told that some users specifically want to type in static networking info when setting up the machine
16:47:02 <bgilbert> red_beard: this isn't just for experimentation, it's for prod installs
16:47:12 <red_beard> i understand
16:47:35 <bgilbert> red_beard: your proposal is to require non-self-hosted mechanisms for that?
16:47:37 <red_beard> it's just giving folks a way to have a process of building the config in a general purpose environment
16:47:48 <darkmuggle> the tl;dr is that users hate catching grub prompts to use kargs.
16:48:07 <dustymabe> darkmuggle: yep
16:48:28 <darkmuggle> The idea for using nmtui was to give people a friendly way to configure it
16:48:51 <red_beard> bgilbert: the proposal is "there will always be endless reasons users think they need to add things to the OS.  in this case it's that they want to treat a deterministic deployment as non-deterministic"
16:48:52 <darkmuggle> Then users don't have to know the interface naming, or have to figure out the ip= karg syntaxxx
16:49:32 <shivarammysore> You still have to present list of network interfaces and allow user to pick which one they want to configure
16:49:49 <red_beard> "so allow users to explore in the more flexible environment, generate their configuration, and then try to use it"
16:49:57 <bgilbert> red_beard: I think "if you want to configure static networking, do it from that OS over there -->" is a tough sell
16:50:24 <red_beard> it's not "in that OS over there", it's making it more obvious how the concerns are separated.
16:50:37 <bgilbert> also, it gives us less control.  they either have to do an extra reboot (boot the Other OS, generate network config, reboot into live ISO for installer)
16:50:47 <bgilbert> or they run coreos-installer from the other OS, where we can't give them nice affordances
16:51:32 <bgilbert> maybe I'm not understanding your proposal
16:52:12 <red_beard> eh... never mind.  no point in fighting the tide.
16:52:18 <dustymabe> red_beard: let's try this
16:52:30 <dustymabe> here is the current proposal 2 which sounds somewhat similar to what you're proposing
16:52:38 <jlebon> red_beard: the live ISO is essentially what you're describing, right? it's a flexible environment in which to play around with settings
16:52:38 <dustymabe> 2: use the live ISO installer environment to allow users to inspect the hardware profile and provide a TUI to allow them to interactively configure networking that can then be passed to the installed system
16:52:49 <dustymabe> red_beard: how would you change that? ^^
16:53:15 <shivarammysore> To build on bgilbert proposal, have a UI to collect network info first and then feed it into Live installer - only if the option to select static network (timeout after 10seconds)
16:53:34 <bgilbert> shivarammysore: yep, that's basically proposal 2
16:53:41 <bgilbert> no timeout though
16:54:11 <bgilbert> the live ISO currently boots to a prompt if no kargs, or to the installer if given coreos.inst.* kargs
16:54:25 <shivarammysore> timeout is specified so that one does not need to use this case in most cases.  Users would not like to press any buttons
16:54:32 <bgilbert> we could change the former to boot to TUI, or have users run the TUI themselves
16:54:39 <bgilbert> shivarammysore: pressing buttons is exactly the problem, though
16:54:51 <bgilbert> a 10-second timeout after a 40-minute BIOS POST is a non-starter
16:54:57 <shivarammysore> agree
16:55:05 <dustymabe> shivarammysore: you can use the live ISO and automate the install already
16:55:18 <jlebon> hmm, one way to bridge the TUI <--> coreos-installer gap is to have a `coreos-installer install --import-nm-keyfiles` which takes the keyfile from the current boot
16:55:18 <dustymabe> we're talking about the case where a user was already doing it interactively
16:55:37 <dustymabe> jlebon: right, that's an implementation detail of proposal 2
16:56:09 <jlebon> dustymabe: right, wanted to talk about the details :)
16:56:11 <bgilbert> jlebon: fun fact, CL coreos-install had a little-used option to do that for networkd
16:56:49 <red_beard> i guess that's fine.  it sounds like more work to me to try to start building those machines with a magic image (the live ISO) as opposed to what's already built Fedora and FCOS
16:56:58 <red_beard> keeping separate packages in each
16:57:07 <jlebon> bgilbert: ohh neat
16:57:07 <bgilbert> the live ISO already exists in FCOS
16:57:49 <bgilbert> we're already working to an installer flow which is more or less: boot live ISO, run coreos-installer
16:58:18 <dustymabe> #link https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/#_installing_from_live_iso
16:59:25 <dustymabe> so the only different I can see between doing what red_beard is proposing and 2 is that we would make sure to keep unneeded packages out of the base
16:59:48 <dustymabe> if we were to include nmtui (we currently don't) then we'd need to add that
17:00:07 <dustymabe> i think if we were going to be included a lot of new software I'd agree more with red_beard
17:00:30 <dustymabe> but if it's small to me it's not worth the completely separate env
17:00:50 <bgilbert> can we ship nmtui in a container image?  :-P
17:01:19 <dustymabe> :), yes but we'd be making the live ISO image much larger to achieve that goal IIUC
17:01:24 <bgilbert> we should look at how much it'll expand the image, because of course we'd ship it everywhere unless we made special provisions
17:01:29 <bgilbert> *base image
17:01:33 <darkmuggle> 250K if memory serves
17:02:02 <bgilbert> including deps?
17:02:30 <dustymabe> it's 3 packages right now:   NetworkManager-tui-1:1.20.10-1.fc31.x86_64
17:02:32 <dustymabe> newt-0.52.21-2.fc31.x86_64
17:02:33 <darkmuggle> we already have the other ones, there's just two depes
17:02:34 <dustymabe> slang-2.3.2-6.fc31.x86_64
17:03:14 <dustymabe> if we really didn't want those we could probably do something crafty just with the live ISO
17:03:33 <dustymabe> though that would also affect live ISO boots that aren't meant for installs
17:03:58 <dustymabe> ok, so it seems like we've had a lot of discussion around `2`
17:04:03 <dustymabe> anyone want to talk about `1` ?
17:04:24 <dustymabe> which is basically "have ignition detect failure and present interactive interface"
17:04:37 <cyberpear> for 1, I assume you'd want nmtui in the initrd?
17:04:45 <dustymabe> cyberpear: or equivalent, yet
17:04:48 <dustymabe> yes*
17:04:50 <cyberpear> so that NM can bring up networking, per the recent merges
17:05:11 <dustymabe> cyberpear: OR detect failure and boot into some sort of live real root enironment (i.e. not the initrd)
17:05:40 <jlebon> my main issue with 1 is that it's an implicit way to try to achieve your goal
17:06:05 <dustymabe> jlebon: right, we would be assuming the user wants an interactive environment
17:06:15 <dustymabe> but there are some cases where you don't need or want networking in the initramfs
17:06:16 <jlebon> imagine the docs saying: "if you want to provide network configuration interactively, just let ignition fail"
17:06:41 <darkmuggle> the idea for 1 is that it would apply for things where the user had interacted or was reasonably assumed to interact with it
17:06:47 <darkmuggle> e.g. `ignition.url=http....`
17:07:20 <dustymabe> darkmuggle: +1 - so cases where we can detect the user had already interacted
17:07:20 <bgilbert> the problem is, in general, it's hard to figure out when we have enough networking configured for Ignition to succeed
17:07:30 <cyberpear> I'd "assume" that networking is by far the most common case, and be fine w/ having a `ignition.no-network` or similar option to skip the nmtui w/o networking
17:07:46 <jlebon> darkmuggle: if you have access to the kernel cmdline, then you could have an explicit karg for opting into it :)
17:08:19 <darkmuggle> and the "failure" would be one the fetch stage
17:08:25 <shivarammysore> users who use NTUI would most likely not know how to handle kargs
17:08:42 <darkmuggle> you would be surprised... kargs are not fun
17:09:02 <darkmuggle> ip= kargs are miserable especially when counting colons
17:09:19 <darkmuggle> regardless, the ask is to provide an ergonomic way
17:09:35 * lorbus notes that he has something for Open Floor
17:09:41 <dustymabe> ok we've gone quite a while on this topic
17:09:51 <dustymabe> let's see if we can take any conclusions away
17:10:00 <shivarammysore> can I ask if we will add whois package?
17:10:15 <dustymabe> shivarammysore: openfloor would be a good time
17:10:20 <lorbus> is the matrix bridge down again? 😅
17:10:20 <dustymabe> coming soon
17:10:28 <dustymabe> lorbus: we heard you
17:10:29 <red_beard> yeah, dustymabe, i think the world i live in and the world FCOS users _want_ to live in are just different and it's time for me to accept that.
17:10:58 <dustymabe> so we have two proposals:
17:11:07 <dustymabe> 1: detect on firstboot (ignition boot) that networking isn't set up and present a TUI to the user
17:11:09 <dustymabe> 2: use the live ISO installer environment to allow users to inspect the hardware profile and provide a TUI to allow them to interactively configure networking that can then be passed to the installed system
17:11:27 <dustymabe> and a possible 3rd, which is to not use FCOS at all for 2 but use Fedora base instead
17:11:48 <dustymabe> the main advantages I can see to #2 over #1 are:
17:12:11 <dustymabe> - we already have live ISO environment for installs where users have an interactive experience for doing an install
17:12:34 <dustymabe> - failure detection in ignition is hard and there are many cases where we'll have to work out exactly what the right behavior should be
17:13:22 <dustymabe> do all of you see those as advantages?
17:13:36 <bgilbert> +1
17:14:02 <shivarammysore> +1
17:14:09 <dustymabe> advantages of #1 over #2:
17:14:09 <jlebon> +1
17:14:37 <dustymabe> - it could allow for a user to correct something that was misconfigured
17:14:51 <cyberpear> so, #2 is "install time network conifg" where #1 is "first boot time network config"
17:15:14 <dustymabe> cyberpear: roughly, yes
17:16:02 <dustymabe> it's not exactly that simple because we do need to take into account how hard it would be to achieve the goal in each implementation
17:16:13 <cyberpear> seems reasonable for all cases except for where machines are built in a different env than where they are deployed, such as an OEM
17:16:31 <bgilbert> cyberpear: true
17:16:54 <dustymabe> yes, that is one limitation
17:17:25 <dustymabe> ok i'll take this summary to the ticket
17:17:33 <cyberpear> or installing the OS while on the network, and deploying to a disconnected network, though not sure how common that is
17:17:54 <dustymabe> cyberpear: DHCP makes life so much simpler :)
17:18:14 <dustymabe> ok will move to open floor soon.
17:18:42 <dustymabe> #info dustymabe will record a summary of this discussion in ticket #427
17:18:44 <cyberpear> agreed on DHCP, though some/many folks still insist on static
17:19:37 <dustymabe> #proposed there are advantages and disadvantes of the two approaches to solve this problem. Right now we'd like to explore the implementation of using the live ISO installer environment to preset a TUI to the user to see if that solves the pain points for our users
17:19:44 <dustymabe> acks/nacks ?
17:20:09 <bgilbert> I'd drop the first sentence, but sure :-)
17:20:19 <shivarammysore> can you check with the user who posted this request if this solution would solve their problem?
17:20:32 <dustymabe> #proposed Right now we'd like to explore the implementation of using the live ISO installer environment to preset a TUI to the user to see if that solves the pain points for our users
17:20:54 <bgilbert> shivarammysore: we've been told that it does
17:20:58 <dustymabe> shivarammysore: i mentioned early on it's actually requested by downstream customers using RHEL CoreOS
17:21:06 <shivarammysore> awesome.  thanks
17:21:12 <cyberpear> +1
17:21:20 <jlebon> s/preset/present/  otherwise ack  :)
17:21:25 <dustymabe> so our management teams have been communicating with us through this process to help give feedback
17:21:48 <shivarammysore> +1
17:21:57 <davdunc> +1 on the jlebon mod.
17:22:03 <bgilbert> +1
17:22:22 <dustymabe> #agreed Right now we'd like to explore the implementation of using the live ISO installer environment to present a TUI to the user to see if that solves the pain points for our users
17:22:36 <dustymabe> we can of course continue to have the discussion in the ticket
17:22:40 <dustymabe> #topic open floor
17:22:47 <dustymabe> lorbus: shivarammysore: your turns!!
17:23:04 <lorbus> I seem to experience lag on the IRC to matrix bridge -- nvmd if you receive this or the following messages at a weird time
17:23:06 <shivarammysore> are we going to add whois package to FCOS?  If so, which release?
17:23:47 <shivarammysore> right now I have a work around.  I would like my production setup to use something without hacks
17:24:01 <bgilbert> shivarammysore: any reason you can't run it in a container?
17:24:06 <dustymabe> shivarammysore: so for whois you're just querying domain name ownership information ?
17:24:07 <cyberpear> what does the whois package do?
17:24:14 <cyberpear> jwhois?
17:24:24 <dustymabe> cyberpear: for example `whois dustymabe.com`
17:24:33 <red_beard> yeah, which is `jwhois`
17:24:35 <shivarammysore> schema verification of JSON/YAML - need whois
17:24:49 <shivarammysore> can't use it on containers.
17:25:15 <jlebon> why not?
17:25:22 <red_beard> i'm curious as well
17:25:32 <dustymabe> shivarammysore: can you give us an example of a json file you're verifying using whois?
17:26:30 <cyberpear> to be clear, we're talking about this one https://src.fedoraproject.org/rpms/whois/blob/master/f/whois.spec ?
17:26:32 <shivarammysore> when you start having lots of containers, management is an issue.  We maintain containers for all serivces
17:26:56 <bgilbert> is there a technical reason it can't run in a container?
17:27:01 <shivarammysore> we don't user K8s.  that is too heavy for us.  we rolled our own
17:27:05 <red_beard> shivarammysore: that's kindof the point though, determinism around deployments
17:27:10 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/432
17:27:10 <red_beard> and you don't need k8s to do that
17:27:17 <bgilbert> in general, with FCOS, we only ship software in the host that must be there
17:27:32 <bgilbert> and everything else should run in a container.
17:27:44 <dustymabe> shivarammysore: are you running *any* containers on FCOS today?
17:27:51 <shivarammysore> the schema is proprietary to us and how we do validation - it should be no different for any
17:28:01 <shivarammysore> oh yes - we run most of the services on containers
17:28:07 <dustymabe> shivarammysore +1
17:28:36 <dustymabe> shivarammysore: you've mentioned before that you came from container linux, did you not have this problem there ?
17:28:37 <shivarammysore> we have a few Go binaries wrapped in systemd units for a few things
17:28:58 <shivarammysore> can't go into details :-(
17:29:35 <bgilbert> okay, so unfortunately I'm not hearing an actionable reason for shipping whois in the host :-(
17:29:37 <cyberpear> what is the disk impact?
17:29:39 <shivarammysore> per the issue, is the objective to use whois-nl package or whois
17:29:56 <red_beard> it's around adding packages which aren't needed to setup the _host_
17:30:13 <cyberpear> "Searches for an object in a RFC 3912 database." -- seems unnecessary to me, except that we already ship one of its sub-packages and layering doesn't work in this case
17:30:14 <dustymabe> shivarammysore: you opened the issue, right. which package did you desire?
17:30:15 <red_beard> workloads run in containers. packages added to the host are to set it up so that it can run containers.
17:30:30 <shivarammysore> dustymabe need whois
17:30:34 <dustymabe> right
17:30:59 <cyberpear> but maybe we should fix the "can't upgrade base packages when installing another package"
17:31:21 <lorbus> before I drop: I wanted to ask on an update on the image upload to GCP and Azure issues (at least for Azure there is at least one open issue wrt fixed VHD file size)
17:31:21 <lorbus> It's not a immediate priority, though. So I'll bring that up again another time.
17:31:21 <lorbus> Bye everyone o/
17:31:26 <dustymabe> ok so I think in general it would be really easy to run this in a container and we don't have any reason to need it in the host. It would be extra container build overhead but in the future package layering should be better
17:31:27 <shivarammysore> cyberpear yes if that is fixed, I can live with it
17:31:54 <dustymabe> bgilbert: I had a question for you in https://github.com/coreos/fedora-coreos-tracker/issues/432#issuecomment-602310619
17:32:01 <dustymabe> I don't even think we need whois-nls in the host
17:32:24 <dustymabe> and it looks like you responded
17:32:26 <bgilbert> yeah, just commented there
17:32:32 <bgilbert> that was an artifact of what I was doing at the time
17:32:40 <bgilbert> I saw that whois-nls got removed by that change, so I added it back
17:32:51 <bgilbert> which I don't think I would have done if whois wasn't also there
17:32:58 <bgilbert> but that was F30, so maybe something pulled it in as a dep
17:33:04 <bgilbert> anyway, I'm +1 to dropping -nls if we're not shipping the base
17:33:05 <dustymabe> bgilbert: ahh, so maybe I should check if anything recommends whois-nls
17:33:37 <dustymabe> shivarammysore: ok I'll try to summarize this in the ticket. we can help you either run it in a container, or try to unblock the package layering flow
17:33:44 * jlebon has to drop -- thanks all!
17:33:59 <shivarammysore> unblock package layering is what I would prefer dustymabe
17:34:10 <dustymabe> shivarammysore: and thanks for helping us work through some early paper cuts in FCOS land
17:34:10 <bgilbert> shivarammysore: but, as an editorial comment: please don't package-layer for that.  layering has some problems and we don't encourage it unless necessary.
17:34:12 <dustymabe> it helps
17:34:31 <bgilbert> shivarammysore: in general, software should run in containers by default
17:34:36 <shivarammysore> Even I would not like to do that, but, sometimes needed
17:34:54 <dustymabe> anything else before I close the meeting out?
17:35:04 <shivarammysore> one more
17:35:13 <shivarammysore> I am still working this ...
17:36:03 <shivarammysore> Create a instance on AWS. Then start a custom systemd service.  Stop the instance, create a AMI and now launch a new Instance with the AMI
17:36:33 <dustymabe> is there an open issue for this?
17:36:35 <shivarammysore> I see that reboot does not happen correctly -to restart service, .  Still investigating .  This is heads up
17:36:45 <bgilbert> note that you'll get duplicate machine ID, duplicate ssh host keys, ...
17:36:48 <shivarammysore> no issue filed yet - just heads up
17:37:11 <shivarammysore> bgilbert totally understand
17:37:30 <dustymabe> yeah it sounds problematic (similar to running packer). we should probably open an issue to discuss this and the problems it entails
17:37:43 <dustymabe> ok I'm going to end the meeting now since we're over time and I've got to run
17:37:48 <dustymabe> #endmeeting