14:00:29 <nils> #startmeeting modularity_wg
14:00:29 <zodbot> Meeting started Tue May 30 14:00:29 2017 UTC.  The chair is nils. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:00:29 <zodbot> The meeting name has been set to 'modularity_wg'
14:00:29 <nils> #meetingtopic Meeting of the Modularity Working Group (once every two weeks)
14:00:29 <nils> #chair dgilmore langdon
14:00:29 <zodbot> Current chairs: dgilmore langdon nils
14:00:36 <nils> #topic Roll Call
14:00:40 <langdon> .hello langdon
14:00:41 <contyk> .hello psabata
14:00:41 <zodbot> langdon: langdon 'Langdon White' <langdon@fishjump.com>
14:00:42 <nils> .hello nphilipp
14:00:44 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
14:00:48 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
14:01:13 <tflink> .hello tflink
14:01:14 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
14:01:15 <sgallagh> .hello sgallagh
14:01:17 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
14:01:17 <jkurik> .hello jkurik
14:01:20 <zodbot> jkurik: jkurik 'Jan Kurik' <jkurik@redhat.com>
14:01:21 <nils> #chair tflink
14:01:21 <zodbot> Current chairs: dgilmore langdon nils tflink
14:01:43 <nils> #chair sct
14:01:43 <zodbot> Current chairs: dgilmore langdon nils sct tflink
14:01:50 <sct> .hello sct
14:01:51 <zodbot> sct: sct 'Stephen Tweedie' <sct@redhat.com>
14:02:17 <nils> #topic Agenda
14:02:38 <nils> Anybody got something for the agenda? The one on the board is empty...
14:02:56 <langdon> sct and i were discussing one if threebean is around?
14:03:08 * langdon not sure why he added the "?"
14:03:43 * contyk is a little distracted
14:04:29 * jkaluza is here
14:05:19 * threebean joins
14:05:50 <langdon> sct, do you want to add the agenda item?
14:06:23 <sct> langdon: I had two or three. :)  Which part did you want to bring up here? Maintaining the core shared dependency lists?
14:06:50 <sct> Or gating addition of new deps?
14:08:25 <langdon> the maintenance part.. so like "what tools in infra can be used to track the deps of 'items' " as we seem to keep collecting over and over
14:08:34 <sct> Right
14:08:40 * nils peddles the agenda board: https://board.net/p/modularity-wg-agendas
14:09:04 <karsten_> .hello karsten_
14:09:05 <zodbot> karsten_: Sorry, but you don't exist
14:09:08 <karsten_> .hello karsten
14:09:11 <zodbot> karsten_: karsten 'Karsten Hopp' <karsten@redhat.com>
14:09:48 <sct> #topic Dependency management and tracking
14:10:26 <sct> OK, so this is something Langdon and I were talking about --- an area we'd like to improve but it's a pretty generic problem, we don't have it broken down yet.
14:10:39 <sct> Modularity depends on a core set of shared dependencies
14:11:03 <sct> and those dependencies need managed; we need to figure which ones to add to the core of the distro, which ones to bundle with modules
14:11:46 <sct> and we need to grow the core dependencies as needed, but in a managed fashion --- if a new app commit pulls in an additional dependency, that should require manual approval and should be flagged to the developer.
14:12:10 <sct> So there's a whole set of tools we could look at --- for reporting dependencies for individual packages and/or entire subsets like Gnome or KDE;
14:12:23 <sct> and for managing the core deps as we commit updates
14:12:41 <sct> and for making sure that the build roots reflect the full set of shared dependencies that we offer.
14:14:09 <sct> So paraphrasing Langdon's natural question above, this is a big and vague problem area, are there are any obvious tools we can build upon in infra to help get started?
14:14:09 <sgallagh> So what sorts of tools are we looking for, exactly?
14:14:35 <sgallagh> Ones to monitor the various modules and notice if a reasonable subset of them have all started pulling in a particular dependency and propose that for inclusion in the common set?
14:15:00 <sct> A reporting tool like that is the obvious first one
14:15:02 <sgallagh> Or are we talking about monitoring the dependencies of individual modules themselves to keep an eye on when they start to grow?
14:15:17 <sgallagh> *grow "upstream" in traditional Fedora, I mean
14:15:23 <sct> the other natural one would be an rpmdiff-type tool to flag up additional dependencies in one version of a build to a new candidate
14:15:39 * contyk returns
14:15:42 <langdon> sorry.. less distracted now..
14:16:38 * contyk nods
14:16:45 <contyk> I think all this is sort of planned
14:16:53 <langdon> sgallagh, i think both of those things.. it just seems like every time i turn around, someone has written a new way to find/manage deps.. would be nice if we had that data captured in one place and then we could track chnage over time, breakage, visualization, etc
14:17:00 <nils> sct, who would be the person to approve the added component?
14:17:15 <contyk> it's expected application modules would follow rpms/* dist-git branches and whenever the branch content is updated, we try rebuilding the module which references that branch
14:17:19 <jkurik> is not PDC the tool for keeping dependencies tracked ?
14:17:23 <contyk> if there's a new dependency that is missing, we'll noticed that
14:17:36 <jkaluza> sct: you can get list of all modules including a component built from particular branch by querying PDC - could be useful when writing tool which tries to find out the subset of shared components between the modules.
14:17:40 <contyk> of course if a dependency disappears, nothing will break and we'll just carry cruft
14:17:44 <contyk> that needs to be manually inspected
14:18:07 <sct> nils: Well, gating and approving would be a good bit further down the road.  We need a proper gating CI pipeline in place before we can add that as an additional test.
14:18:20 <langdon> contyk, well.. we want not just the maintainer to notice.. we want the infra to flag it up too in case other people need to know and so fesco knows when things are growing
14:18:28 <sct> nils: For now I'd settle for sending a developer a report with a new build that flags up any grown dependencies.
14:18:30 <contyk> anyway, if a new version of a package that is to be included in the module is a missing a dependency and the rebuild fails, the modules maintainer should get in touch with the packager and coordinate; is it really needed? why?
14:19:22 <contyk> langdon: there's no way we would automatically add a dependency to a module, it's a conscious task
14:19:52 <jkaluza> jkurik: This is example of query to find out what modules include RPM from particular branch: https://pdc.fedoraproject.org/rest_api/v1/unreleasedvariants/?component_name=net-tools&component_branch=f26&fields=variant_uid
14:20:04 <nils> sct, but the developer put in the new dep in the first place, I suppose deliberately. This sounds a little like "Are you sure you want to do what you just instructed me to do?"
14:20:14 <sgallagh> contyk: It's a conscious task on the part of the packager/module-owner, but knowing that things are growing is useful from a larger perspective
14:20:27 <contyk> sgallagh: yes, of course
14:20:29 <nils> If that requires the developer to sign off on their own change, that is.
14:20:32 <contyk> just saying it cannot happen unnoticed
14:20:38 <sct> nils: Not really
14:20:51 <sgallagh> contyk: Sure, but it can be handwaved through without thought
14:21:24 <sct> nils: Dependencies grow for all sorts of reasons, sometimes because something else you are using grew a new dependency of its own... the top-level app developer doesn't necessarily even know.
14:21:28 <langdon> contyk, like we don't want to tell them they have to send an email announcing the change.. we want the infra to capture it and then be able to do "useful" things based on the event
14:21:31 <nils> sct, right now, we don't seem to pull in deps of deps automatically, i.e. every dep that sneaks in has to go past the (module) maintainer.
14:21:41 <jkaluza> Info that "things are growing" can be also obtained from PDC, btw... you can compare previous versions of modules to see how many rpms it includes
14:21:55 <nils> We'd have a lot less trouble if it weren't that way ;)I
14:21:59 <sct> and I'd like to be able to balance the burden here
14:21:59 <langdon> nils, yes.. by design ;) but we would like to automate it under certain conditions
14:22:07 <nils> langdon, ahh
14:22:30 <contyk> we have so much data; you can generate all kinds of reports :)
14:22:43 <nils> langdon, so there are plans so that I could say "add component XYZ and all its deps that aren't satisfied by any of the module deps"`
14:22:44 <nils> ?
14:22:48 <langdon> nils, basically it is done that way to solve the problem sct mentioned above .. but it doesn't need to be manual
14:22:56 <nils> good to hear
14:22:59 <langdon> contyk, reports!
14:23:15 * langdon prefers printed and fed-ex'd
14:23:44 <ignatenkobrain> .hello ignatenkobrain
14:23:45 <zodbot> ignatenkobrain: ignatenkobrain 'Igor Gnatenko' <ignatenko@redhat.com>
14:23:53 <langdon> nils, the "plan" there is basically the module dep checker dominika has been working on.. just a sub-set use case
14:24:12 <jkaluza> langdon: is there source code for that?
14:25:09 <langdon> jkaluza, yes? but my hand waving does not always lead to easy links to code ;)
14:25:30 <jkaluza> langdon: ah ignore, you mean for generating the dependencies... I thought about generating reports
14:25:44 * jkaluza did not switch the context fast enough
14:26:40 <langdon> jkaluza, ah.. gotcha.. well, asamalik also has a visualizer.. and bpo did some of it.. so.. we have pieces.. but I guess what I am asking here, oh factory-2 team, where is that "actually" gonna live and can we start making those changes
14:26:55 <langdon> so people stop writing one off scripts and we actually build it in the right place
14:27:42 <ignatenkobrain> hopefully it is going to be properly written ;)
14:27:54 <langdon> ignatenkobrain, pshaw
14:28:18 <ignatenkobrain> I strongly recommend whoever will be writing *any* dependency resolution thing to talk with us (dnf team)
14:28:45 <langdon> ignatenkobrain, ohh.. most of them i know of use dnf to do the "actual"  resolution.. its just wrappers
14:28:57 <ignatenkobrain> langdon: no ;)_
14:29:17 <nils> langdon, ignatenkobrain can give you any number of ways to do that wrong with dnf :)
14:29:22 <langdon> ha
14:29:45 <contyk> so hopefully this new depchase ignatenkobrain is working on will help us all
14:29:52 <ignatenkobrain> nils: langdon: most common is to do queries for package Requires and then query for something what provides this X
14:29:54 <contyk> but it's not something langdon wants in the future, I think
14:30:02 <langdon> well.. any which way, i think that is an easy problem to solve.. i think the "plan" is harder..
14:30:04 <contyk> it won't help you with monitoring dependency chain changes within modules
14:30:22 <ignatenkobrain> in 99% cases it is wrong ;)
14:30:27 <ignatenkobrain> contyk: yeah, I know
14:30:32 <nils> yeah, the question is not "what is the dep", more "where is it"
14:30:48 <langdon> and why is it here now
14:30:50 <ignatenkobrain> I want also something better (like something what asamalik started doing)
14:31:12 <ignatenkobrain> nils: to know *where* you need to know *what*
14:31:22 <ignatenkobrain> and by queries you most probably will find wrong *what*
14:31:34 <nils> ignatenkobrain, listening to you, that problem is solved, we only need to use the right solution.
14:31:36 <ignatenkobrain> however, in many cases it works (but slow and not in all cases)
14:32:04 <langdon> so.. threebean, jkaluza do we know where this goes?
14:32:13 <nils> ignatenkobrain, and to be honest, we can live with imperfect solutions for a while there :o)
14:32:17 <ignatenkobrain> nils: yep -- https://github.com/fedora-modularity/depchase/blob/libsolv/depchase.py#L308-L321
14:32:51 * jkaluza is reading backlog
14:33:09 <ignatenkobrain> anyway, if you want to do something quick -- I'm perfectly fine whichever things you do, but if we want to think about future -- we need to do it properly
14:33:41 <ignatenkobrain> I would love to see some visualisation grouped by existing modules and etc.
14:34:09 <ignatenkobrain> with ability to do things like "here we have alternatives, what will happen if I will choose different one?"
14:34:13 <langdon> ignatenkobrain, it seems like this is a common issue, has dnf team thought about just offering a simple api that "works correctly" for people to use when trying to resolve?
14:35:38 <ignatenkobrain> langdon: when DNF has begun, no one thought about flexibility... There *is* some API which you could use for depsolving.. like dnf.Base.install("xxx") and then dnf.Transaction.list_installs() or such
14:36:08 <ignatenkobrain> for getting runtime requirements it is more than enough, for build-time it is not because thing is way more complicated
14:36:32 <langdon> ignatenkobrain, ahh.. ok..
14:36:33 <ignatenkobrain> I'm here also to find what kind of flexibility is needed and plan that for DNF 3.0 rewrite
14:37:00 <nils> ignatenkobrain, if that doesn't lead us too far, what is that much more complicated about build requirements?
14:37:54 <sgallagh> ignatenkobrain: Wait, DNF is getting another *rewrite*?
14:38:04 <contyk> once a year
14:38:06 <ignatenkobrain> sgallagh: it didn't have any rewrite yet
14:38:14 <langdon> microdnf
14:38:17 <sgallagh> ...
14:38:20 <ignatenkobrain> nils: if you want recursive BuildRequires, you are in trouble because many times you can't install dependencies for 2 source packages at the same time which means you have to do things iteratively which means you can't take advantage of auto-minimizing and checking decisions and etc.
14:38:45 <ignatenkobrain> langdon: it is very minimal wrapper around libdnf which is going to have "as minimum functionality as it can"
14:39:08 <sgallagh> ignatenkobrain: Well, hopefully the recursive buildrequires won't be a long-term problem.
14:39:12 <jkaluza> langdon: Well, the data about the built modules are in PDC (like for charts showing how many rpms the module includes over time, or even the dep tree of what module depends on what module or if the modules/dependencies are up to date)
14:39:14 <sgallagh> That's more of a bootstrapping issue
14:39:20 <nils> ignatenkobrain, yeah, the recursion is the key point, right?
14:39:22 <jkaluza> langdon: but we do not have anything displaying those in some nice way
14:39:31 <sgallagh> Getting things into the build infrastructure for the first time.
14:40:11 <contyk> it will still be useful for packaging new modules
14:40:34 <jkaluza> langdon: I can help people with getting the data out of PDC or even changing the PDC or MBS so the data will be there, but not with writing the service showing the data
14:40:42 <ignatenkobrain> nils: you need to queue new src packages after resolving runtime requires for src packages (like a 10 iterations for bootstrap module)
14:41:40 <langdon> sct, i think we could move on, yes?
14:42:55 <sct> langdon: Yes, I think we've spent enough time on it!
14:43:27 <nils> sct, langdon, anything you want to #info?
14:43:32 <langdon> i was wondering if we should talk about shared userspace..
14:43:36 <langdon> nils, i do
14:43:38 <langdon> one sec
14:45:01 <langdon> ok? #info jkaluza & threebean to formally respond with where to put dependency information for rpms and modules so that we can start to build up tools around the information (tracking change, visualization, etc)
14:45:29 <contyk> isn't that an action?
14:45:38 <langdon> contyk, i was on the fence
14:45:57 <langdon> slight wording change:
14:45:58 <langdon> ok? #info jkaluza & threebean to formally respond with where dependency information for rpms and modules is and will be so that we can start to build up tools around the information (tracking change, visualization, etc)
14:45:59 <jkaluza> isn't that all in PDC?
14:46:05 <langdon> is it?
14:46:05 <contyk> it is
14:46:12 <contyk> but you need to answer this formally
14:46:14 <contyk> ;)
14:46:15 <threebean> :p
14:46:16 <langdon> ha
14:46:21 <langdon> you don't if that is "true"
14:46:24 <jkaluza> I said it multiple times here :)
14:46:32 <sgallagh> Can I suggest that maybe this would be a really good blog post?
14:46:38 <jkaluza> [16:39] <jkaluza> langdon: Well, the data about the built modules are in PDC (like for charts showing how many rpms the module includes over time, or even the dep tree of what module depends on what module or if the modules/dependencies are up to date)
14:46:46 <langdon> i thought pdc just had the information about the composes themselves.. not the traceback on rpm deps
14:46:48 <sgallagh> "How to monitor and retrieve module dependency data from PDC"
14:46:59 <contyk> sgallagh: "How we get things done in Modularity WG"
14:47:09 * langdon entirely missed that line by jkaluza
14:47:14 <sgallagh> contyk: No, I don't think we want to advertise that :-P
14:47:26 <jkaluza> [16:40] <jkaluza> langdon: I can help people with getting the data out of PDC or even changing the PDC or MBS so the data will be there, but not with writing the service showing the data
14:47:32 <jkaluza> langdon: ^ that one followed
14:47:34 <langdon> yeah.. caught that
14:48:05 <langdon> ok.. so would visualization type stuff go in pdc as well?
14:48:13 <langdon> or separate but fed from the data there?
14:48:18 <jkaluza> no, PDC is just database with an API to store the data
14:48:21 <threebean> separate, but fed from pdc.
14:48:24 <jkaluza> ^ that
14:48:32 <langdon> rigth.. ok.. so basically BPO..
14:48:37 <jkaluza> langdon: yes
14:48:49 <jkaluza> but we have much more data in PDC than in the BPO times
14:49:10 <langdon> does infra plan to include/support bpo (or similar) in the near future?
14:49:13 <sgallagh> Sorry, BPO?
14:49:29 <contyk> sgallagh: yet another service for monitoring things
14:49:34 <jkaluza> langdon: I think it's not on our current TODO list in near time, threebean would know :)
14:49:35 <contyk> sgallagh: "build pipeline overview"
14:49:40 <sgallagh> ok
14:49:51 <langdon> jkaluza, no.. that is YASMT.. ;)
14:49:53 <contyk> it was presented at Flock in Krakow and then died
14:50:08 <contyk> like the next day
14:50:18 <sgallagh> ah
14:50:22 <langdon> sgallagh, its meant to (originally) visualization the status of a module in infra
14:50:34 <langdon> but it has similar UX problems to dnf.. like how do we represent modules
14:50:40 <langdon> and .. got stuck on that
14:50:54 <langdon> *visualize
14:51:20 <langdon> ok.. so..
14:51:24 <threebean> ok, this is not on my roadmap for anytime in the near future.  like jkaluza says, the data should be there.
14:51:52 <contyk> and we're collecting more and more data over time
14:51:57 <langdon> ?? #info PDC has all the dependency information needed to do things like track module changes, visualize deps, etc
14:52:11 <contyk> yes
14:52:20 <jkaluza> langdon: you can add me as a contact if someone wants to show the data and how to access them
14:52:27 <langdon> threebean, but if someone else built "YA BPO" ;) .. would infra be able to support it?
14:52:40 <contyk> yes
14:52:40 <langdon> #info PDC has all the dependency information needed to do things like track module changes, visualize deps, etc
14:52:53 <contyk> anybody can query PDC and listen for fedmsg
14:53:00 <langdon> contyk, yes to what? ^^
14:53:15 <contyk> langdon: to your last question for threebean :)
14:53:32 <langdon> contyk, ha.. thats not a good enough answer.. ;)
14:53:32 <jkaluza> contyk: I have no idea if f2 is going to support bpo, that's up to threebean I think
14:53:54 <threebean> langdon: it depends on the details of YA BPO implementation.
14:53:58 <jkaluza> contyk: even when written by someone else :)
14:54:07 * langdon points out it is in the diagram .. therefore, it must be true
14:54:09 <threebean> if someone goes to write a quick visualization, have them loop me and jkaluza in for advice early on.
14:54:28 <threebean> on current f2.0 schedules, there will be no "proper BPO" before f27.
14:54:37 <dgilmore> sorry I am late and too absent
14:54:39 <jkaluza> langdon: It can also run on fed-mod.org for some time and still be helpful
14:54:39 <langdon> #info modularity team will continue to consider some visualization tool for modules
14:54:57 <contyk> dgilmore: we were just talking about how you will support a new, undefined service
14:55:02 <langdon> jkaluza, right
14:55:11 <langdon> dgilmore, in node.js!
14:55:15 <dgilmore> contyk: sure, let me go find 100 volunteers to do it :)
14:56:23 <langdon> ok.. i think we got what we needed.. will see if I can get someone to work on "new BPO"
14:56:39 <contyk> langdon: to show it at flock
14:56:44 <jkaluza> :D
14:56:46 <contyk> langdon: and then kill it
14:56:47 <langdon> lol.. good idea!
14:56:56 <threebean> (for real, see if you can narrow that scope a bit...)
14:56:58 <langdon> and maybe some build triggered scripts to do before/after checking
14:57:59 <langdon> threebean, yeah.. i have it in my head i think.. need to write it down.. sct & I keep having trouble talking to people about the deps/relationships because it is too abstract.. so I am looking to solve that problem
14:58:15 <langdon> plus the problem of not allowing things to "slip in"
14:59:20 <langdon> ok.. i think that horse is definitely dead
14:59:25 <langdon> should we wrap it up?
14:59:35 * dgilmore wonders if he can do three meetings at once
14:59:47 * langdon knows dgilmore is very talented
14:59:56 <langdon> nils, ^^
15:00:36 <nils> langdon, sure. Do we have anything else besides that? Something for open floor?
15:00:50 <langdon> nils, well.. we are at time already
15:00:56 <langdon> and I know i have another
15:00:58 <langdon> meeting
15:01:01 <nils> langdon, right :)
15:01:15 <nils> in that case, if anything was missed, put it on next meeting's agenda ;)
15:01:20 <nils> thanks everybody!
15:01:21 <dgilmore> langdon: :D not sure I am that talented
15:01:27 <nils> #endmeeting