Linux XIA - merge proposal

Linux XIA - merge proposal

[Michel Machado]

Hi there,

    We have been developing Linux XIA, a new network stack that 
emphasizes evolvability and interoperability, for a couple of years, and 
it has now reached a degree of maturity that allows others to experiment 
with it.  In addition to the kernel implementation, we have a userland 
tool to set up the stack, a Wireshark fork with extensions for XIA 
packets, an implemented solution to interoperate with IPv4 networks, 
forwarding performance evaluation of our code, support for Linux 
containers, and an exemplifying demo; all documented on our wiki [1]. 
Outlining the full benefits of XIA goes beyond the scope of this e-mail, 
but we present an in-depth discussion on our wiki and in our 
peer-reviewed academic publications referenced there.

    Linux XIA is not yet mature enough for production, but we do think 
that broader exposure will help us to move forward faster, and get more 
in tune with the interested community. Thus, we are writing to ask to 
merge our work into the Linux kernel.

    We are aware of all of the effort required to review a large chunk 
of code, so to lower this burden, we propose to merge our new network 
stack via staging. This would give time for the community at large to 
get acquainted with XIA's concepts, and review the code. Of course, we 
will address feedback as it comes in. We're aware that the current Linux 
staging process [2] is geared toward drivers and filesystems, but not 
network stacks. We're fine with clearly marking Linux XIA as being under 
staging as well as helping to define this review process for network stacks.

    Linux XIA is highly confined in its folder net/xia. There are a few 
headers in folder include/net, but all the names of these headers are of 
the form xia_*.h. Patches outside these files are minimal. This high 
isolation is a guarantee that Linux XIA can be dropped out of the kernel 
with little work if needed.

    Finally, there are several other research groups around the world 
working on other future Internet architectures.  We hope that the 
process of merging Linux XIA will provide a template for these other 
groups to merge their network stacks into Linux in the future.

    Thank you very much for your time,


[1] https://github.com/AltraMayor/XIA-for-Linux/wiki
[2] http://www.kroah.com/log/linux/linux-staging-update.html

-- 
[ ]'s
Michel Machado
on behalf of the Linux XIA team at Boston University

Re: Linux XIA - merge proposal

[Eric W. Biederman]

Michel Machado <michel@digirati.com.br> writes:

>    Linux XIA is not yet mature enough for production, but we do think
> that broader exposure will help us to move forward faster, and get
> more in tune with the interested community. Thus, we are writing to
> ask to merge our work into the Linux kernel.

I took a quick glance at the code to see if could figure out what
you were talking about.   Especialy as I could not find any documents
on the packet format.  The code does not transmit packets.  All of the
packet transmit functions are stubs.  I can think of no greater failure
for a networking protocol.

Since the code does nothing except have buzz-word compliant descriptions
that are content free I respectfully suggest you make it mostly work
before foisting it on people.

My advice for a future day when you have code that can transmit packets
is to start small.  Have patches that do things and do them clearly.
Once you have a minimal version of working code merged then work your
way up.

For just sharing your code you might simply want to have your own tree
that you publish publicly.

Eric

Re: Linux XIA - merge proposal

[Daniel Borkmann]

On 03/03/2015 06:29 PM, Michel Machado wrote:
...
>     We're fine with clearly marking Linux XIA as being under staging as well as helping to define this review process for network stacks.

With regard to staging, the code there is usually horrible and I'm
not sure anyone really looks there, that would mitigate the review
problem to the time when you try to get it out from there, so I'm
not sure it brings anything. ;)

+1 on what Eric said, would have also been nice if you had clearly
described in your mail (w/o buzz words) what it is and what it does.

Are you trying to introduce a new network stack as an alternative
to the current one, e.g. something like FreeBSD's netgraph?

Re: Linux XIA - merge proposal

[Michel Machado]

Hi Eric,

>>     Linux XIA is not yet mature enough for production, but we do think
>> that broader exposure will help us to move forward faster, and get
>> more in tune with the interested community. Thus, we are writing to
>> ask to merge our work into the Linux kernel.
>
> I took a quick glance at the code to see if could figure out what
> you were talking about.   Especialy as I could not find any documents
> on the packet format.  The code does not transmit packets.  All of the
> packet transmit functions are stubs.  I can think of no greater failure
> for a networking protocol.

    I don't know what led you this wrong conclusion, but please let me 
show that is not the case. Section 3.3 of our original paper linked 
below describes the forwarding algorithm as well as the header format.

    https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final13.pdf

    One can find the reference to this paper in our XIA 101 page in our 
wiki:

https://github.com/AltraMayor/XIA-for-Linux/wiki/XIA-101

    The code does transmit packet, you can find function 
__xip_fill_in_hdr() in file net/xia/output.c 
(https://github.com/AltraMayor/XIA-for-Linux/blob/xia/net/xia/output.c) 
that fills out the XIP header, and examples to how set up some 
experiments with our stack are available here:

https://github.com/AltraMayor/XIA-for-Linux/wiki/zFilter-principal
https://github.com/AltraMayor/XIA-for-Linux/wiki/4ID-principals
https://github.com/AltraMayor/XIA-for-Linux/wiki/An-exemplifying-demo

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Michel Machado]

Hi Daniel,

>> We're fine with clearly marking Linux XIA as being under staging
>> as well as helping to define this review process for network stacks.
>
> With regard to staging, the code there is usually horrible and I'm
> not sure anyone really looks there, that would mitigate the review
> problem to the time when you try to get it out from there, so I'm
> not sure it brings anything. ;)

    I suggested staging as a way to incrementally review the code. XIA 
is very modular, each component is a kernel module. So one can review a 
kernel module at a time. My intention is to lower the review burden as 
much as I can. I'm open to suggestions to work this out.

> +1 on what Eric said, would have also been nice if you had clearly
> described in your mail (w/o buzz words) what it is and what it does.

    XIA is not an incremental design over IP, and a reasonable 
explanation would've made my e-mail exceeding large. Our wiki has a lot 
content explaining the design as well as demoing it, and I can answer 
questions here, or through another communication medium that you guys 
feel comfortable.

> Are you trying to introduce a new network stack as an alternative
> to the current one, e.g. something like FreeBSD's netgraph?

    I'm trying to introduce a new network stack that enables one to 
implement network designs that are incompatible with IP for a reason or 
another. On top of that, these designs, once ported to Linux XIA, could 
be combined to work together as we tried to highlight in the following demo:

https://github.com/AltraMayor/XIA-for-Linux/wiki/An-exemplifying-demo

    There're a good number of designs in the literature that hasn't seen 
the sun light because they are hard to experiment with since they are 
not compatible with IP. Our hope is to bring these designs to life and 
have a true evaluation of them. That is, we hope that Linux XIA will 
enable the community at large to crowdsource the future Internet. We 
work this vision out in this technical report:

http://www.cs.bu.edu/techreports/pdf/2015-001-linux-xia.pdf

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Greg KH]

On Tue, Mar 03, 2015 at 07:31:58PM -0500, Michel Machado wrote:
> Hi Daniel,
> 
> >>We're fine with clearly marking Linux XIA as being under staging
> >>as well as helping to define this review process for network stacks.
> >
> >With regard to staging, the code there is usually horrible and I'm
> >not sure anyone really looks there, that would mitigate the review
> >problem to the time when you try to get it out from there, so I'm
> >not sure it brings anything. ;)
> 
>    I suggested staging as a way to incrementally review the code.

Yet you failed to at least ask the maintainer of the staging portion of
the kernel, if this is a valid use of staging, and his resources...

Not nice :(

greg k-h

Re: Linux XIA - merge proposal

[Michel Machado]

On 03/04/2015 12:49 AM, Greg KH wrote:
> On Tue, Mar 03, 2015 at 07:31:58PM -0500, Michel Machado wrote:
>> Hi Daniel,
>>
>>>> We're fine with clearly marking Linux XIA as being under staging
>>>> as well as helping to define this review process for network stacks.
>>>
>>> With regard to staging, the code there is usually horrible and I'm
>>> not sure anyone really looks there, that would mitigate the review
>>> problem to the time when you try to get it out from there, so I'm
>>> not sure it brings anything. ;)
>>
>>     I suggested staging as a way to incrementally review the code.
>
> Yet you failed to at least ask the maintainer of the staging portion of
> the kernel, if this is a valid use of staging, and his resources...
>
> Not nice :(
>
> greg k-h

Hi Greg,

    I am sorry for having not asked you. Given that the staging process 
doesn't include network stacks, I'd thought that the correct way to go 
would be to bring the discussion to netdev first. Please understand that 
I'm not gaming, I'm just following my best understanding of the 
documentation of how to submit code. I'm really in uncharted waters here 
since all my accepted patches are tiny.

    In addition, I intend to work on the feedback I get. I'm not 
planning to have the code upstream and run away.

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Alex Elsayed]

Michel Machado wrote:

> Hi there,
> 
>     We have been developing Linux XIA, a new network stack that
> emphasizes evolvability and interoperability, for a couple of years, and
> it has now reached a degree of maturity that allows others to experiment
> with it.

>From looking at your wiki, "network stack" may have been a poor choice of 
term - it looks like rather than being a new network stack (which in Linux, 
is commonly used to refer to the software stack that lives between the APIs 
and the hardware), this is a new protocol (and framework _for_ protocols) 
operating at the same level of the network as IP, with ideas extending 
upwards through TCP.

Now, that's a rather different proposal - witness that RDS, TIPC, etc all 
made it into the kernel relatively easily, especially when compared to 
netmap, or any other system that tried to replace the Linux networking 
infrastructure.

Re: Linux XIA - merge proposal

[Michel Machado]

 >>    We have been developing Linux XIA, a new network stack that
 >> emphasizes evolvability and interoperability, for a couple of years,
 >> and it has now reached a degree of maturity that allows others to
 >> experiment with it.
 >
 > From looking at your wiki, "network stack" may have been a poor
 > choice of term - it looks like rather than being a new network
 > stack (which in Linux, is commonly used to refer to the software
 > stack that lives between the APIs and the hardware), this is a
 > new protocol (and framework _for_ protocols) operating at
 > the same level of the network as IP, with ideas extending
 > upwards through TCP.
 >
 > Now, that's a rather different proposal - witness that RDS, TIPC,
 > etc all made it into the kernel relatively easily, especially when
 > compared to netmap, or any other system that tried to replace
 > the Linux networking infrastructure.

Hi Alex,

    Thank you very much for having caught this definition mismatch! I 
personally see TCP/IP as a network stack on its own right. From this 
perspective, XIA, TCP/IP, RDS, and TIPC are all at the same level as you 
concluded. I don't mind adopting the definitions that are more common 
among kernel developers.

    Linux XIA is not replacing the other protocols (using your 
definition). Linux XIA is another protocol, and, as you correctly 
pointed out: a framework for other protocols.

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Bjorn Helgaas]

On Tue, Mar 3, 2015 at 11:29 AM, Michel Machado <michel@digirati.com.br> wrote:
> Hi there,
>
>    We have been developing Linux XIA, a new network stack that emphasizes
> evolvability and interoperability, for a couple of years, and it has now
> reached a degree of maturity that allows others to experiment with it.  In
> addition to the kernel implementation, we have a userland tool to set up the
> stack, a Wireshark fork with extensions for XIA packets, an implemented
> solution to interoperate with IPv4 networks, forwarding performance
> evaluation of our code, support for Linux containers, and an exemplifying
> demo; all documented on our wiki [1]. Outlining the full benefits of XIA
> goes beyond the scope of this e-mail, but we present an in-depth discussion
> on our wiki and in our peer-reviewed academic publications referenced there.
>
>    Linux XIA is not yet mature enough for production, but we do think that
> broader exposure will help us to move forward faster, and get more in tune
> with the interested community. Thus, we are writing to ask to merge our work
> into the Linux kernel.
>
>    We are aware of all of the effort required to review a large chunk of
> code, so to lower this burden, we propose to merge our new network stack via
> staging. This would give time for the community at large to get acquainted
> with XIA's concepts, and review the code. Of course, we will address
> feedback as it comes in. We're aware that the current Linux staging process
> [2] is geared toward drivers and filesystems, but not network stacks. We're
> fine with clearly marking Linux XIA as being under staging as well as
> helping to define this review process for network stacks.
>
>    Linux XIA is highly confined in its folder net/xia. There are a few
> headers in folder include/net, but all the names of these headers are of the
> form xia_*.h. Patches outside these files are minimal. This high isolation
> is a guarantee that Linux XIA can be dropped out of the kernel with little
> work if needed.
>
>    Finally, there are several other research groups around the world working
> on other future Internet architectures.  We hope that the process of merging
> Linux XIA will provide a template for these other groups to merge their
> network stacks into Linux in the future.

>From my point of view (far outside the networking world and completely
incompetent to judge the merits of XIA), this email seems like a
reasonable starting point for a discussion, and I'm disappointed that
it got shut down so fast.  It might not be in *exactly* the form
people are looking for, but hey, it's an introductory email, and
Michel seems open to constructive feedback.

Bjorn

Re: Linux XIA - merge proposal

[Eric Dumazet]

On Tue, 2015-03-10 at 12:01 -0500, Bjorn Helgaas wrote:

> From my point of view (far outside the networking world and completely
> incompetent to judge the merits of XIA), this email seems like a
> reasonable starting point for a discussion, and I'm disappointed that
> it got shut down so fast.  It might not be in *exactly* the form
> people are looking for, but hey, it's an introductory email, and
> Michel seems open to constructive feedback.

I never got initial mail on netdev, only replies to it, including part
of initial message ?

This might explain some people never commented, maybe ?

Re: Linux XIA - merge proposal

[Bjorn Helgaas]

On Tue, Mar 10, 2015 at 12:19 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Tue, 2015-03-10 at 12:01 -0500, Bjorn Helgaas wrote:
>
>> From my point of view (far outside the networking world and completely
>> incompetent to judge the merits of XIA), this email seems like a
>> reasonable starting point for a discussion, and I'm disappointed that
>> it got shut down so fast.  It might not be in *exactly* the form
>> people are looking for, but hey, it's an introductory email, and
>> Michel seems open to constructive feedback.
>
> I never got initial mail on netdev, only replies to it, including part
> of initial message ?
>
> This might explain some people never commented, maybe ?

Here it is, in case it's useful: https://lkml.org/lkml/2015/3/3/602

Re: Linux XIA - merge proposal

[Michel Machado]

On 03/10/2015 01:01 PM, Bjorn Helgaas wrote:
>  From my point of view (far outside the networking world and completely
> incompetent to judge the merits of XIA), this email seems like a
> reasonable starting point for a discussion, and I'm disappointed that
> it got shut down so fast.  It might not be in *exactly* the form
> people are looking for, but hey, it's an introductory email, and
> Michel seems open to constructive feedback.

    Thank you for the support, Bjorn.

    I've brainstormed with friends questions that could address the 
concerns that haven't been voiced in this thread, and I answered those 
questions in our FAQ page:

    https://github.com/AltraMayor/XIA-for-Linux/wiki/FAQ

    While I would like to have Linux XIA merged, I also don't want to 
become importunate. So, if nobody else shows interest in this thread, 
I'm going to leave it to rest.

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Greg KH]

On Fri, Mar 13, 2015 at 11:26:46AM -0400, Michel Machado wrote:
> On 03/10/2015 01:01 PM, Bjorn Helgaas wrote:
> > From my point of view (far outside the networking world and completely
> >incompetent to judge the merits of XIA), this email seems like a
> >reasonable starting point for a discussion, and I'm disappointed that
> >it got shut down so fast.  It might not be in *exactly* the form
> >people are looking for, but hey, it's an introductory email, and
> >Michel seems open to constructive feedback.
> 
>    Thank you for the support, Bjorn.
> 
>    I've brainstormed with friends questions that could address the concerns
> that haven't been voiced in this thread, and I answered those questions in
> our FAQ page:
> 
>    https://github.com/AltraMayor/XIA-for-Linux/wiki/FAQ
> 
>    While I would like to have Linux XIA merged, I also don't want to become
> importunate. So, if nobody else shows interest in this thread, I'm going to
> leave it to rest.

But you have yet to actually post the code, why give up now, you haven't
even tried.  A number of people have given you hints here as to what you
should be doing / trying, to just run away without even attempting it
seems quite odd and defeatist.

good luck,

greg k-h

Re: Linux XIA - merge proposal

[Oded Gabbay]

On 03/15/2015 11:37 AM, Greg KH wrote:
> On Fri, Mar 13, 2015 at 11:26:46AM -0400, Michel Machado wrote:
>> On 03/10/2015 01:01 PM, Bjorn Helgaas wrote:
>>>  From my point of view (far outside the networking world and completely
>>> incompetent to judge the merits of XIA), this email seems like a
>>> reasonable starting point for a discussion, and I'm disappointed that
>>> it got shut down so fast.  It might not be in *exactly* the form
>>> people are looking for, but hey, it's an introductory email, and
>>> Michel seems open to constructive feedback.
>>
>>     Thank you for the support, Bjorn.
>>
>>     I've brainstormed with friends questions that could address the concerns
>> that haven't been voiced in this thread, and I answered those questions in
>> our FAQ page:
>>
>>     https://github.com/AltraMayor/XIA-for-Linux/wiki/FAQ
>>
>>     While I would like to have Linux XIA merged, I also don't want to become
>> importunate. So, if nobody else shows interest in this thread, I'm going to
>> leave it to rest.
>
> But you have yet to actually post the code, why give up now, you haven't
> even tried.  A number of people have given you hints here as to what you
> should be doing / trying, to just run away without even attempting it
> seems quite odd and defeatist.
>
> good luck,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
Hi Michel,

Although I don't know you and I'm not familiar with XIA, I would like to 
encourage you to listen to Greg and continue with your efforts to upstream XIA.

I can tell you, from my recent experience of upstreaming AMD's HSA driver to drm 
(amdkfd), that I understand your response/frustration. However, you need to 
understand that the feedback you got is not because the maintainers/devs don't 
want your code. It is because they want your code to be in the best shape it can 
be before merging it to the kernel. Once it is in the kernel, it is distributed 
to millions of people and it is maintained practically forever.

 From my experience, the major changes that make your code better are usually 
done *before* upstreaming the code. Once the maintainers/devs realize that the 
code has reached some level of compliance with kernel standards, and the code 
can't get any better by leaving it outside the kernel, I'm sure they will accept 
the code. Especially if they will see that you are dedicated to improving it.

Please see this enlightening post from drm maintainer, Dave Airlie, about trying 
to push code to upstream kernel:
http://airlied.livejournal.com/80112.html

It is called:
"you have a long road to walk, but first you have to leave the house (or why 
publishing code is STEP ZERO)."

BTW, I believe I was one of the reasons he wrote that post (it was published 
shortly after I published amdkfd's code) :)

So bottom line, keep working on it. People gave you pointers what to do. Follow 
that and don't give up. In the end, I'm sure you will be very happy once the 
code is merged into the mainline kernel.

Good luck,

	Oded

Re: Linux XIA - merge proposal

[Michel Machado]

On 03/15/2015 05:37 AM, Greg KH wrote:
> On Fri, Mar 13, 2015 at 11:26:46AM -0400, Michel Machado wrote:
>> On 03/10/2015 01:01 PM, Bjorn Helgaas wrote:
>>>  From my point of view (far outside the networking world and completely
>>> incompetent to judge the merits of XIA), this email seems like a
>>> reasonable starting point for a discussion, and I'm disappointed that
>>> it got shut down so fast.  It might not be in *exactly* the form
>>> people are looking for, but hey, it's an introductory email, and
>>> Michel seems open to constructive feedback.
>>
>>     Thank you for the support, Bjorn.
>>
>>     I've brainstormed with friends questions that could address the concerns
>> that haven't been voiced in this thread, and I answered those questions in
>> our FAQ page:
>>
>>     https://github.com/AltraMayor/XIA-for-Linux/wiki/FAQ
>>
>>     While I would like to have Linux XIA merged, I also don't want to become
>> importunate. So, if nobody else shows interest in this thread, I'm going to
>> leave it to rest.
>
> But you have yet to actually post the code, why give up now, you haven't
> even tried.  A number of people have given you hints here as to what you
> should be doing / trying, to just run away without even attempting it
> seems quite odd and defeatist.

Hi Greg,

    After reading your e-mail, I realized that I hadn't really 
understood the feedback that I've received. I'm going to follow your 
advice. Figuring out how to break our codebase in series of patches is 
going to take sometime, but I intend to get it moving.

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Michel Machado]

On 03/15/2015 06:03 AM, Oded Gabbay wrote:
> Hi Michel,
>
> Although I don't know you and I'm not familiar with XIA, I would like to
> encourage you to listen to Greg and continue with your efforts to
> upstream XIA.
>
> I can tell you, from my recent experience of upstreaming AMD's HSA
> driver to drm (amdkfd), that I understand your response/frustration.
> However, you need to understand that the feedback you got is not because
> the maintainers/devs don't want your code. It is because they want your
> code to be in the best shape it can be before merging it to the kernel.
> Once it is in the kernel, it is distributed to millions of people and it
> is maintained practically forever.
>
>  From my experience, the major changes that make your code better are
> usually done *before* upstreaming the code. Once the maintainers/devs
> realize that the code has reached some level of compliance with kernel
> standards, and the code can't get any better by leaving it outside the
> kernel, I'm sure they will accept the code. Especially if they will see
> that you are dedicated to improving it.
>
> Please see this enlightening post from drm maintainer, Dave Airlie,
> about trying to push code to upstream kernel:
> http://airlied.livejournal.com/80112.html
>
> It is called:
> "you have a long road to walk, but first you have to leave the house (or
> why publishing code is STEP ZERO)."
>
> BTW, I believe I was one of the reasons he wrote that post (it was
> published shortly after I published amdkfd's code) :)
>
> So bottom line, keep working on it. People gave you pointers what to do.
> Follow that and don't give up. In the end, I'm sure you will be very
> happy once the code is merged into the mainline kernel.
>
> Good luck,
>
>      Oded

Hi Oded,

    Thank you for sharing your experience. Dave Airlie's perspective 
helps. I'm going to walk the road "with a little help from friends."

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Oded Gabbay]

On 03/17/2015 02:25 PM, Michel Machado wrote:
> On 03/15/2015 05:37 AM, Greg KH wrote:
>> On Fri, Mar 13, 2015 at 11:26:46AM -0400, Michel Machado wrote:
>>> On 03/10/2015 01:01 PM, Bjorn Helgaas wrote:
>>>>  From my point of view (far outside the networking world and completely
>>>> incompetent to judge the merits of XIA), this email seems like a
>>>> reasonable starting point for a discussion, and I'm disappointed that
>>>> it got shut down so fast.  It might not be in *exactly* the form
>>>> people are looking for, but hey, it's an introductory email, and
>>>> Michel seems open to constructive feedback.
>>>
>>>     Thank you for the support, Bjorn.
>>>
>>>     I've brainstormed with friends questions that could address the
>>> concerns
>>> that haven't been voiced in this thread, and I answered those
>>> questions in
>>> our FAQ page:
>>>
>>>     https://github.com/AltraMayor/XIA-for-Linux/wiki/FAQ
>>>
>>>     While I would like to have Linux XIA merged, I also don't want to
>>> become
>>> importunate. So, if nobody else shows interest in this thread, I'm
>>> going to
>>> leave it to rest.
>>
>> But you have yet to actually post the code, why give up now, you haven't
>> even tried.  A number of people have given you hints here as to what you
>> should be doing / trying, to just run away without even attempting it
>> seems quite odd and defeatist.
>
> Hi Greg,
>
>     After reading your e-mail, I realized that I hadn't really
> understood the feedback that I've received. I'm going to follow your
> advice. Figuring out how to break our codebase in series of patches is
> going to take sometime, but I intend to get it moving.
>
> [ ]'s
> Michel Machado
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Hi Michel,
If you want an example of how to break a fairly medium+ 
code-base/driver, I believe amdkfd is a good example. Originally, I had 
dumped 80+ patches, but after getting some flames & help from drm devs, 
I rearranged them into a nicely 25 patches:

1. Several patches that implements helper functions in other drivers 
(mm, iommu, radeon)
2. Patch to add IOCTL definitions (kfd_ioctl.h)
3. Patch to add skeleton code base of driver
4. Several patches that add different modules in the amdkfd driver
5. Several patches that implement the IOCTLs in the amdkfd drivers

See this link for the final set that was merged into 3.19-rc1: 
http://cgit.freedesktop.org/~gabbayo/linux/?h=amdkfd-v6

Take a look from commit "Merge branch 'drm-next-3.19'  to the HEAD of 
the log.

If you need additional advice, you can contact me.

	Oded

Re: Linux XIA - merge proposal

[Michel Machado]

On 03/18/2015 06:27 AM, Oded Gabbay wrote:
 > Hi Michel,
 > If you want an example of how to break a fairly medium+
 > code-base/driver, I believe amdkfd is a good example. Originally, I had
 > dumped 80+ patches, but after getting some flames & help from drm devs,
 > I rearranged them into a nicely 25 patches:
 >
 > 1. Several patches that implements helper functions in other drivers
 > (mm, iommu, radeon)
 > 2. Patch to add IOCTL definitions (kfd_ioctl.h)
 > 3. Patch to add skeleton code base of driver
 > 4. Several patches that add different modules in the amdkfd driver
 > 5. Several patches that implement the IOCTLs in the amdkfd drivers
 >
 > See this link for the final set that was merged into 3.19-rc1:
 > http://cgit.freedesktop.org/~gabbayo/linux/?h=amdkfd-v6
 >
 > Take a look from commit "Merge branch 'drm-next-3.19'  to the HEAD of
 > the log.
 >
 > If you need additional advice, you can contact me.
 >
 >      Oded

Hi Oded,

    Thank you for your example. After thinking on how to get our 
repository as a set of patches, it became clear that I have to start 
with a new repository and gradually move the code from our current 
repository to this new one. Because the commits of the current 
repository follows how we arrived where we are now, and this makes the 
goal of only reviewing the final code harder.

    I'm going to take advantage that Linux XIA is broken down into 
kernel modules, and have a serie of patches for each module. With the 
exception of the core module, on which all other modules depend, all 
modules don't have dependencies between them. The problem I'm running 
into with this approach is that some patches will be large. For example, 
the core module has these files:

$ wc net/xia/*.c
    744   2246  15523 net/xia/dag.c
    897   2378  23006 net/xia/fib.c
    360    931   8240 net/xia/fib_frontend.c
     72    169   1486 net/xia/locktbl.c
    106    259   2116 net/xia/main.c
    281    858   7241 net/xia/output.c
   1471   4573  37952 net/xia/route.c
    598   1579  13783 net/xia/socket.c
    109    305   2852 net/xia/vxidty.c
   4638  13298 112199 total

    I can try to break these large patches into smaller ones, but this 
would likely break code coherence since each module is supposed to 
self-contained. My understanding is that you created skeletons to 
gradually fill them in to address similar issue, but I'm not sure how to 
do it in my case.

    I'll keep pounding on this task. Any advice is more than welcome.

    By the way, the new repository will be based on David Miller's -next 
networking tree:

http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git

[ ]'s
Michel Machado

Re: Linux XIA - merge proposal

[Bjorn Helgaas]

[+cc Arnd]

On Fri, Mar 20, 2015 at 1:46 PM, Michel Machado <michel@digirati.com.br> wrote:
> On 03/18/2015 06:27 AM, Oded Gabbay wrote:
>> Hi Michel,
>> If you want an example of how to break a fairly medium+
>> code-base/driver, I believe amdkfd is a good example. Originally, I had
>> dumped 80+ patches, but after getting some flames & help from drm devs,
>> I rearranged them into a nicely 25 patches:
>>
>> 1. Several patches that implements helper functions in other drivers
>> (mm, iommu, radeon)
>> 2. Patch to add IOCTL definitions (kfd_ioctl.h)
>> 3. Patch to add skeleton code base of driver
>> 4. Several patches that add different modules in the amdkfd driver
>> 5. Several patches that implement the IOCTLs in the amdkfd drivers
>>
>> See this link for the final set that was merged into 3.19-rc1:
>> http://cgit.freedesktop.org/~gabbayo/linux/?h=amdkfd-v6
>>
>> Take a look from commit "Merge branch 'drm-next-3.19'  to the HEAD of
>> the log.
>>
>> If you need additional advice, you can contact me.
>>
>>      Oded
>
> Hi Oded,
>
>    Thank you for your example. After thinking on how to get our repository
> as a set of patches, it became clear that I have to start with a new
> repository and gradually move the code from our current repository to this
> new one. Because the commits of the current repository follows how we
> arrived where we are now, and this makes the goal of only reviewing the
> final code harder.

I agree that people likely won't want to review the entire trail of
how you got to where you are now.

>    I'm going to take advantage that Linux XIA is broken down into kernel
> modules, and have a serie of patches for each module. With the exception of
> the core module, on which all other modules depend, all modules don't have
> dependencies between them. The problem I'm running into with this approach
> is that some patches will be large. For example, the core module has these
> files:
>
> $ wc net/xia/*.c
>    744   2246  15523 net/xia/dag.c
>    897   2378  23006 net/xia/fib.c
>    360 931 8240 net/xia/fib_frontend.c
>     72    169   1486 net/xia/locktbl.c
>    106    259   2116 net/xia/main.c
>    281 858 7241 net/xia/output.c
>   1471   4573  37952 net/xia/route.c
>    598   1579  13783 net/xia/socket.c
>    109    305   2852 net/xia/vxidty.c
>   4638  13298 112199 total
>
>    I can try to break these large patches into smaller ones, but this would
> likely break code coherence since each module is supposed to self-contained.

We try very hard to make sure that the kernel is buildable and
functional after every commit, even if that commit is part of a larger
series.  This helps people bisect to find regressions.  Just something
to keep in mind when breaking things up.

> My understanding is that you created skeletons to gradually fill them in to
> address similar issue, but I'm not sure how to do it in my case.

This problem sounds similar to that of adding a new processor
architecture.  I cc'd Arnd, since he's helped do that several times.
Here's how one recent architecture was added (from [1]):

  nios2: Build infrastructure
  nios2: Assembly macros and definitions
  nios2: Kernel booting and initialization
  nios2: Exception handling
  nios2: Traps exception handling
  nios2: Memory management
  nios2: I/O Mapping
  nios2: MMU Fault handling
  nios2: Page table management
  nios2: Process management
  nios2: Cache handling
  nios2: TLB handling
  nios2: Interrupt handling
  nios2: DMA mapping API
  nios2: ELF definitions
  nios2: System calls handling
  nios2: Signal handling support
  nios2: Library functions
  nios2: Device tree support
  nios2: Time keeping
  nios2: Cpuinfo handling
  nios2: Miscellaneous header files
  nios2: Nios2 registers
  nios2: Module support
  nios2: ptrace support
  Add ELF machine define for Nios2
  MAINTAINERS: Add nios2 maintainer
  Documentation: Add documentation for Nios2 architecture

Bjorn

[1] http://lkml.kernel.org/r/1397824031-4892-1-git-send-email-lftan@altera.com