perf fu

Change-Id: I5f2acfe14d3dbffa96aab00870b094a9486f414e

COPYING

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.

Makefile.am

@@ -19,7 +19,6 @@ SUBDIRS = \
 BUILT_SOURCES = $(top_srcdir)/.version
 EXTRA_DIST = \
 	     .version \
-	     contrib/osmo-upf.spec.in \
 	     debian \
 	     git-version-gen \
 	     osmoappdesc.py \

README.md

@@ -5,16 +5,16 @@ Homepage
 --------
 
 The official homepage of the project is
-https://osmocom.org/projects/osmoupf/wiki
+https://osmocom.org/projects/osmo-upf/wiki
 
 GIT Repository
 --------------
 
 You can clone from the official osmo-upf.git repository using
 
-	git clone git://git.osmocom.org/osmo-upf.git
+	git clone https://gitea.osmocom.org/cellular-infrastructure/osmo-upf
 
-There is a cgit interface at https://git.osmocom.org/osmo-upf/
+There is a web interface at https://gitea.osmocom.org/cellular-infrastructure/osmo-upf.
 
 To submit patches, see "Contributing" below.
 

configure.ac

@@ -36,16 +36,11 @@ fi
 PKG_PROG_PKG_CONFIG([0.20])
 
 dnl checks for libraries
-AC_SEARCH_LIBS([dlopen], [dl dld], [LIBRARY_DL="$LIBS";LIBS=""])
-AC_SUBST(LIBRARY_DL)
-
-PKG_CHECK_MODULES(LIBOSMOCORE, libosmocore >= 1.5.0)
-PKG_CHECK_MODULES(LIBOSMOVTY, libosmovty >= 1.5.0)
-PKG_CHECK_MODULES(LIBOSMOCTRL, libosmoctrl >= 1.5.0)
-PKG_CHECK_MODULES(LIBOSMOGTLV, libosmo-gtlv >= 0.1.0)
+PKG_CHECK_MODULES(LIBOSMOCORE, libosmocore >= 1.6.0)
+PKG_CHECK_MODULES(LIBOSMOVTY, libosmovty >= 1.6.0)
+PKG_CHECK_MODULES(LIBOSMOCTRL, libosmoctrl >= 1.6.0)
 PKG_CHECK_MODULES(LIBOSMOPFCP, libosmo-pfcp >= 0.1.0)
 PKG_CHECK_MODULES(LIBGTPNL, libgtpnl >= 1.2.0)
-PKG_CHECK_MODULES(LIBNFTNL, libnftnl >= 1.2.1)
 PKG_CHECK_MODULES(LIBNFTABLES, libnftables >= 1.0.2)
 
 dnl checks for header files
@@ -128,7 +123,7 @@ if test "x$enable_ext_tests" = "xyes" ; then
 	fi
 	AC_CHECK_PROG(OSMOTESTEXT_CHECK,osmotestvty.py,yes)
 	 if test "x$OSMOTESTEXT_CHECK" != "xyes" ; then
-		AC_MSG_ERROR([Please install git://osmocom.org/python/osmo-python-tests to run the VTY/CTRL tests.])
+		AC_MSG_ERROR([Please install https://gitea.osmocom.org/cellular-infrastructure/osmo-python-tests to run the VTY/CTRL tests.])
 	fi
 fi
 AC_MSG_CHECKING([whether to enable VTY/CTRL tests])
@@ -203,15 +198,16 @@ AC_OUTPUT(
     include/Makefile
     include/osmocom/Makefile
     include/osmocom/upf/Makefile
+    include/osmocom/pfcptool/Makefile
     src/Makefile
     src/osmo-upf/Makefile
     src/osmo-pfcp-tool/Makefile
     tests/Makefile
     tests/atlocal
+    tests/unique_ids/Makefile
     doc/Makefile
     doc/examples/Makefile
     doc/manuals/Makefile
-    doc/charts/Makefile
     contrib/Makefile
     contrib/systemd/Makefile
     Makefile)

contrib/jenkins.sh

@@ -50,8 +50,12 @@ build_from_netfilter() {
 	git clone "git://git.netfilter.org/$project" "$project"
 	cd "$project"
 	autoreconf --install --force
-	./configure --prefix="$inst/stow/$project" --without-cli --disable-man-doc
-	$MAKE install
+	./configure \
+		--prefix="$inst/stow/$project" \
+		--without-cli \
+		--disable-man-doc \
+		--enable-python=no
+	$MAKE $PARALLEL_MAKE install
 	STOW_DIR="$inst/stow" stow --restow $project
 }
 build_from_netfilter libnftnl

contrib/osmo-pfcp-tool-scripts/osmo-pfcp-tool.cfg

@@ -1,4 +1,10 @@
 log stderr
+ logging color 1
+ logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
+ logging print file basename last
+ logging print level 1
  logging level set-all info
 
 local-addr 127.0.0.2

contrib/osmo-pfcp-tool-scripts/osmo-upf-11.cfg

@@ -1,11 +1,11 @@
 log stderr
  logging filter all 1
  logging color 1
- logging print level 1
- logging print category 1
  logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
  logging print file basename last
- logging print extended-timestamp 1
+ logging print level 1
  logging level set-all notice
  logging level set-all info
  logging level session debug
@@ -21,7 +21,5 @@ ctrl
 timer pfcp x24 5000
 pfcp
  local-addr 127.0.0.11
-gtp
+tunend
  dev create apn11 127.0.0.11
-nft
- table-name osmo-upf-11

contrib/osmo-pfcp-tool-scripts/osmo-upf-12.cfg

@@ -1,11 +1,11 @@
 log stderr
  logging filter all 1
  logging color 1
- logging print level 1
- logging print category 1
  logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
  logging print file basename last
- logging print extended-timestamp 1
+ logging print level 1
  logging level set-all notice
  logging level set-all info
  logging level session debug
@@ -21,7 +21,5 @@ ctrl
 timer pfcp x24 5000
 pfcp
  local-addr 127.0.0.12
-gtp
- dev create apn12 127.0.0.12
-nft
+tunmap
  table-name osmo-upf-12

contrib/osmo-pfcp-tool-scripts/session_est_without_assoc.vty

@@ -1,4 +1,6 @@
 timer pfcp x23 0
 pfcp-peer 127.0.0.1
- session endecaps
+ session tunend
+  ue ip 127.127.127.127
+  gtp access remote f-teid 127.0.0.127 127
   tx session-est-req

contrib/osmo-pfcp-tool-scripts/session_mod.vty

@@ -3,12 +3,14 @@ pfcp-peer 127.0.0.1
  tx assoc-setup-req
  sleep 1
  session
+  ue ip 127.127.127.127
+  gtp access remote f-teid 127.0.0.127 127
   tx session-est-req drop
   sleep 3
-  tx session-mod-req forw
+  tx session-mod-req far forw
   sleep 5
-  tx session-mod-req drop
+  tx session-mod-req far drop
   sleep 3
-  tx session-mod-req forw
+  tx session-mod-req far forw
   sleep 3
   tx session-del-req

contrib/osmo-pfcp-tool-scripts/tunend_plus_tunmap.vty

@@ -1,4 +1,4 @@
-# ACCESS                HOP                           CORE
+# ACCESS                UPF tunmap                    UPF tunend (CORE)
 #                       session 23 = tunmap           session 42 = encaps/decaps
 # GTP 127.0.0.13        127.0.0.12                    127.0.0.11
 # TEID l:23 r:123 <---> r:23 l:123 | l:142 r:42 <---> r:142 l:42 | 192.168.100.42
@@ -11,10 +11,10 @@ timer pfcp x23 0
 pfcp-peer 127.0.0.11
  tx assoc-setup-req
  sleep 1
- session endecaps 42
+ session tunend 42
   ue ip 192.168.100.42
-  gtp access ip 127.0.0.12
-  gtp access teid local 42 remote 142
+  gtp access local f-teid 127.0.0.11 42
+  gtp access remote f-teid 127.0.0.12 142
   tx session-est-req
  sleep 1
 
@@ -22,9 +22,9 @@ pfcp-peer 127.0.0.12
  tx assoc-setup-req
  sleep 1
  session tunmap 23
-  gtp core ip 127.0.0.11
-  gtp core teid local 142 remote 42
-  gtp access ip 127.0.0.13
-  gtp access teid local 123 remote 23
+  gtp core remote f-teid 127.0.0.11 42
+  gtp core local f-teid 127.0.0.12 142
+  gtp access local f-teid 127.0.0.12 123
+  gtp access remote f-teid 127.0.0.13 23
   tx session-est-req
  sleep 1

contrib/osmo-pfcp-tool-scripts/tunend_session_est.vty

@@ -2,7 +2,10 @@ timer pfcp x23 0
 pfcp-peer 127.0.0.1
  tx assoc-setup-req
  sleep 1
- session endecaps
+ session tunend
+  ue ip 127.127.127.127
+  gtp access local f-teid choose
+  gtp access remote f-teid 127.0.0.12 142
   tx session-est-req forw
   sleep 5
   tx session-del-req

contrib/osmo-pfcp-tool-scripts/tunmap_session_est.vty

@@ -3,6 +3,10 @@ pfcp-peer 127.0.0.1
  tx assoc-setup-req
  sleep 1
  session tunmap
+  gtp core remote f-teid 127.0.0.11 42
+  gtp core local f-teid choose
+  gtp access local f-teid choose
+  gtp access remote f-teid 127.0.0.13 23
   tx session-est-req
   sleep 5
   tx session-del-req

contrib/osmo-upf.spec.in

@@ -1,87 +0,0 @@
-#
-# spec file for package osmo-upf
-#
-# Copyright (c) 2017, Martin Hauke <mardnh@gmx.de>
-#
-# All modifications and additions to the file contributed by third parties
-# remain the property of their copyright owners, unless otherwise agreed
-# upon. The license for this file, and modifications and additions to the
-# file, is the same license as for the pristine package itself (unless the
-# license for the pristine package is not an Open Source License, in which
-# case the license is the MIT License). An "Open Source License" is a
-# license that conforms to the Open Source Definition (Version 1.9)
-# published by the Open Source Initiative.
-
-## Disable LTO for now since it breaks compilation of the tests
-## https://osmocom.org/issues/4113
-%define _lto_cflags %{nil}
-
-Name:           osmo-upf
-Version:        @VERSION@
-Release:        0
-Summary:        OsmoUPF: Osmocom User Plane Function
-License:        AGPL-3.0-or-later AND GPL-2.0-or-later
-Group:          Hardware/Mobile
-URL:            https://osmocom.org/projects/osmoupf
-Source:         %{name}-%{version}.tar.xz
-BuildRequires:  autoconf-archive
-BuildRequires:  automake >= 1.9
-BuildRequires:  libtool >= 2
-BuildRequires:  lksctp-tools-devel
-BuildRequires:  pkgconfig >= 0.20
-%if 0%{?suse_version}
-BuildRequires:  systemd-rpm-macros
-%endif
-BuildRequires:  pkgconfig(libgtpnl) >= 1.2.0
-BuildRequires:  pkgconfig(libosmocore) >= 1.6.0
-BuildRequires:  pkgconfig(libosmoctrl) >= 1.6.0
-BuildRequires:  pkgconfig(libosmovty) >= 1.6.0
-BuildRequires:  pkgconfig(talloc)
-%{?systemd_requires}
-
-%description
-OsmoUPF: Osmocom User Plane Function
-
-%prep
-%setup -q
-
-%build
-echo "%{version}" >.tarball-version
-autoreconf -fi
-%configure \
-  --docdir=%{_docdir}/%{name} \
-  --with-systemdsystemunitdir=%{_unitdir}
-make %{?_smp_mflags}
-
-%install
-%make_install
-
-%if 0%{?suse_version}
-%preun
-%service_del_preun %{name}.service
-
-%postun
-%service_del_postun %{name}.service
-
-%pre
-%service_add_pre %{name}.service
-
-%post
-%service_add_post %{name}.service
-%endif
-
-%check
-make %{?_smp_mflags} check || (find . -name testsuite.log -exec cat {} +)
-
-%files
-%license COPYING
-%doc AUTHORS README.md
-%{_bindir}/osmo-upf
-%dir %{_docdir}/%{name}/examples
-%dir %{_docdir}/%{name}/examples/osmo-upf
-%{_docdir}/%{name}/examples/osmo-upf/osmo-upf.cfg
-%dir %{_sysconfdir}/osmocom
-%config(noreplace) %{_sysconfdir}/osmocom/osmo-upf.cfg
-%{_unitdir}/%{name}.service
-
-%changelog

contrib/perf/gen_bpftrace.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+
+import sys
+
+args = sys.argv[1:]
+
+output = []
+
+lib = None
+
+probes = set()
+
+for arg in args:
+	if arg.startswith('/'):
+		lib = arg
+		continue
+	func = arg
+
+	#identifier = '%s-%s' % (lib, func)
+	identifier = func
+	identifier = ''.join([c if c.isalpha() else '_' for c in identifier])
+	identifier.replace('__', '_')
+
+	probes.add((lib, func, identifier))
+
+
+for lib, func, identifier in probes:
+	output.append(
+f'''uprobe:{lib}:{func} {{
+        @{identifier}__count ++;
+        @{identifier}__start = nsecs;
+}}
+uretprobe:{lib}:{func} {{
+        ${identifier}__duration_ns = (nsecs - @{identifier}__start);
+        @{identifier}__total_ns += ${identifier}__duration_ns;
+        @{identifier}__total_ms = @{identifier}__total_ns / 1000000;
+
+        ${identifier}__duration_us = ${identifier}__duration_ns / 1000;
+        @{identifier}__us = hist(${identifier}__duration_us);
+}}''')
+
+output.append(
+'''interval:s:3 {
+        printf("==========================================================");
+        time("%Y-%m-%d %H:%M:%S\\n");''')
+
+for lib, func, identifier in probes:
+	output.append(f'        print(@{identifier}__count);');
+	output.append(f'        print(@{identifier}__total_ms);');
+	output.append(f'        print(@{identifier}__us);');
+
+output.append('}')
+
+print('\n'.join(output))

contrib/perf/nft_cache_release.txt

@@ -0,0 +1,49 @@
+==========================================================2024-05-05 19:16:44
+@nft_evaluate__us:
+[1K, 2K)             111 |@@@@                                                |
+[2K, 4K)             333 |@@@@@@@@@@@@                                        |
+[4K, 8K)             589 |@@@@@@@@@@@@@@@@@@@@@@                              |
+[8K, 16K)           1382 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[16K, 32K)           585 |@@@@@@@@@@@@@@@@@@@@@@                              |
+
+@nft_evaluate__count: 3000
+@nft_cache_update__us:
+[512, 1K)             21 |                                                    |
+[1K, 2K)             138 |@@@@@                                               |
+[2K, 4K)             324 |@@@@@@@@@@@@                                        |
+[4K, 8K)             597 |@@@@@@@@@@@@@@@@@@@@@@                              |
+[8K, 16K)           1395 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[16K, 32K)           525 |@@@@@@@@@@@@@@@@@@@                                 |
+
+@nft_cache_update__count: 3000
+@nft_cache_is_complete__us:
+[2, 4)               974 |@@@@@@@@@@@@@@@@@@@@@@@@@                           |
+[4, 8)              1983 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[8, 16)               26 |                                                    |
+[16, 32)              17 |                                                    |
+
+@nft_cache_is_complete__count: 3000
+@nft_cache_is_updated__us:
+[2, 4)              1684 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[4, 8)              1285 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@             |
+[8, 16)               18 |                                                    |
+[16, 32)              10 |                                                    |
+
+@nft_cache_is_updated__count: 2997
+@nft_cache_release__us:
+[32, 64)              15 |                                                    |
+[64, 128)             38 |@                                                   |
+[128, 256)            88 |@@@@                                                |
+[256, 512)           171 |@@@@@@@@                                            |
+[512, 1K)            192 |@@@@@@@@@                                           |
+[1K, 2K)             498 |@@@@@@@@@@@@@@@@@@@@@@@@                            |
+[2K, 4K)            1065 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[4K, 8K)             852 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@           |
+[8K, 16K)             81 |@@@                                                 |
+
+@nft_cache_release__count: 3000
+
+
+
+
+==========================================================2024-05-05 19:16:47

contrib/perf/nft_cache_update.txt

@@ -0,0 +1,53 @@
+==========================================================2024-05-05 19:10:38
+@nft_evaluate__us:
+[512, 1K)             21 |                                                    |
+[1K, 2K)             264 |@@@@                                                |
+[2K, 4K)             634 |@@@@@@@@@@@                                         |
+[4K, 8K)            1397 |@@@@@@@@@@@@@@@@@@@@@@@@@                           |
+[8K, 16K)           2844 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[16K, 32K)           837 |@@@@@@@@@@@@@@@                                     |
+[32K, 64K)             3 |                                                    |
+
+@nft_evaluate__count: 6000
+@nft_cache_filter_init__us:
+[128, 256)          5985 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[256, 512)             9 |                                                    |
+[512, 1K)              6 |                                                    |
+
+@nft_cache_filter_init__count: 6000
+@nft_cache_evaluate__us:
+[2, 4)               536 |@@@@@                                               |
+[4, 8)              5237 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[8, 16)              150 |@                                                   |
+[16, 32)              77 |                                                    |
+
+@nft_cache_evaluate__count: 6000
+@nft_cache_update__us:
+[512, 1K)            107 |@                                                   |
+[1K, 2K)             289 |@@@@@                                               |
+[2K, 4K)             651 |@@@@@@@@@@@                                         |
+[4K, 8K)            1408 |@@@@@@@@@@@@@@@@@@@@@@@@@                           |
+[8K, 16K)           2828 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[16K, 32K)           714 |@@@@@@@@@@@@@                                       |
+[32K, 64K)             3 |                                                    |
+
+@nft_cache_update__count: 6000
+@nft_cache_filter_fini__us:
+[32, 64)            5841 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[64, 128)            135 |@                                                   |
+[128, 256)             0 |                                                    |
+[256, 512)             0 |                                                    |
+[512, 1K)              0 |                                                    |
+[1K, 2K)               0 |                                                    |
+[2K, 4K)              21 |                                                    |
+[4K, 8K)               3 |                                                    |
+
+@nft_cache_filter_fini__count: 6000
+@nft_cmd_expand__us:
+[1]                  720 |@                                                   |
+[2, 4)             33583 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
+[4, 8)              1551 |@@                                                  |
+[8, 16)               83 |                                                    |
+[16, 32)              63 |                                                    |
+
+@nft_cmd_expand__count: 36000

contrib/perf/now.bt

@@ -0,0 +1,139 @@
+uprobe:/usr/local/lib/libnftables.*:flowtable_free {
+        @flowtable_free__count ++;
+        @flowtable_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:flowtable_free {
+        $flowtable_free__duration_ns = (nsecs - @flowtable_free__start);
+        @flowtable_free__total_ns += $flowtable_free__duration_ns;
+        @flowtable_free__total_ms = @flowtable_free__total_ns / 1000000;
+
+        $flowtable_free__duration_us = $flowtable_free__duration_ns / 1000;
+        @flowtable_free__us = hist($flowtable_free__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:free_const {
+        @free_const__count ++;
+        @free_const__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:free_const {
+        $free_const__duration_ns = (nsecs - @free_const__start);
+        @free_const__total_ns += $free_const__duration_ns;
+        @free_const__total_ms = @free_const__total_ns / 1000000;
+
+        $free_const__duration_us = $free_const__duration_ns / 1000;
+        @free_const__us = hist($free_const__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:chain_free {
+        @chain_free__count ++;
+        @chain_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:chain_free {
+        $chain_free__duration_ns = (nsecs - @chain_free__start);
+        @chain_free__total_ns += $chain_free__duration_ns;
+        @chain_free__total_ms = @chain_free__total_ns / 1000000;
+
+        $chain_free__duration_us = $chain_free__duration_ns / 1000;
+        @chain_free__us = hist($chain_free__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:cache_free {
+        @cache_free__count ++;
+        @cache_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:cache_free {
+        $cache_free__duration_ns = (nsecs - @cache_free__start);
+        @cache_free__total_ns += $cache_free__duration_ns;
+        @cache_free__total_ms = @cache_free__total_ns / 1000000;
+
+        $cache_free__duration_us = $cache_free__duration_ns / 1000;
+        @cache_free__us = hist($cache_free__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:scope_release {
+        @scope_release__count ++;
+        @scope_release__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:scope_release {
+        $scope_release__duration_ns = (nsecs - @scope_release__start);
+        @scope_release__total_ns += $scope_release__duration_ns;
+        @scope_release__total_ms = @scope_release__total_ns / 1000000;
+
+        $scope_release__duration_us = $scope_release__duration_ns / 1000;
+        @scope_release__us = hist($scope_release__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:set_free {
+        @set_free__count ++;
+        @set_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:set_free {
+        $set_free__duration_ns = (nsecs - @set_free__start);
+        @set_free__total_ns += $set_free__duration_ns;
+        @set_free__total_ms = @set_free__total_ns / 1000000;
+
+        $set_free__duration_us = $set_free__duration_ns / 1000;
+        @set_free__us = hist($set_free__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:handle_free {
+        @handle_free__count ++;
+        @handle_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:handle_free {
+        $handle_free__duration_ns = (nsecs - @handle_free__start);
+        @handle_free__total_ns += $handle_free__duration_ns;
+        @handle_free__total_ms = @handle_free__total_ns / 1000000;
+
+        $handle_free__duration_us = $handle_free__duration_ns / 1000;
+        @handle_free__us = hist($handle_free__duration_us);
+}
+uprobe:/usr/local/lib/libnftables.*:obj_free {
+        @obj_free__count ++;
+        @obj_free__start = nsecs;
+}
+uretprobe:/usr/local/lib/libnftables.*:obj_free {
+        $obj_free__duration_ns = (nsecs - @obj_free__start);
+        @obj_free__total_ns += $obj_free__duration_ns;
+        @obj_free__total_ms = @obj_free__total_ns / 1000000;
+
+        $obj_free__duration_us = $obj_free__duration_ns / 1000;
+        @obj_free__us = hist($obj_free__duration_us);
+}
+uprobe:/usr/lib/x86_64-linux-gnu/libc.*:free {
+        @free__count ++;
+        @free__start = nsecs;
+}
+uretprobe:/usr/lib/x86_64-linux-gnu/libc.*:free {
+        $free__duration_ns = (nsecs - @free__start);
+        @free__total_ns += $free__duration_ns;
+        @free__total_ms = @free__total_ns / 1000000;
+
+        $free__duration_us = $free__duration_ns / 1000;
+        @free__us = hist($free__duration_us);
+}
+interval:s:3 {
+        printf("==========================================================");
+        time("%Y-%m-%d %H:%M:%S\n");
+        print(@flowtable_free__count);
+        print(@flowtable_free__total_ms);
+        print(@flowtable_free__us);
+        print(@free_const__count);
+        print(@free_const__total_ms);
+        print(@free_const__us);
+        print(@chain_free__count);
+        print(@chain_free__total_ms);
+        print(@chain_free__us);
+        print(@cache_free__count);
+        print(@cache_free__total_ms);
+        print(@cache_free__us);
+        print(@scope_release__count);
+        print(@scope_release__total_ms);
+        print(@scope_release__us);
+        print(@set_free__count);
+        print(@set_free__total_ms);
+        print(@set_free__us);
+        print(@handle_free__count);
+        print(@handle_free__total_ms);
+        print(@handle_free__us);
+        print(@obj_free__count);
+        print(@obj_free__total_ms);
+        print(@obj_free__us);
+        print(@free__count);
+        print(@free__total_ms);
+        print(@free__us);
+}

contrib/perf/osmo-pfcp-tool.cfg

@@ -0,0 +1,8 @@
+log stderr
+ logging level set-all error
+
+timer pfcp x23 5
+timer pfcp x24 5000
+
+local-addr 127.0.0.2
+listen

contrib/perf/osmo-upf.cfg

@@ -0,0 +1,27 @@
+log stderr
+ logging filter all 1
+ logging color 1
+ logging print level 1
+ logging print category 1
+ logging print category-hex 0
+ logging print file basename last
+ #logging print timestamp date
+ #logging level set-all error
+ logging level set-all info
+
+line vty
+ bind 127.0.0.1
+ctrl
+ bind 127.0.0.1
+
+# PFCP retransmissions
+timer pfcp x23 0
+# PFCP response timeout
+timer pfcp x24 0
+
+pfcp
+ local-addr 127.0.0.1
+tunmap
+ table-name osmo-upf
+tunend
+ dev create apn23

contrib/perf/tunend.vty

@@ -0,0 +1,7 @@
+pfcp-peer 127.0.0.1
+ tx assoc-setup-req
+ sleep 1
+ date
+ n 10000 session create tunend
+ wait responses
+ date

contrib/perf/tunmap.vty

@@ -0,0 +1,7 @@
+pfcp-peer 127.0.0.1
+ tx assoc-setup-req
+ sleep 1
+ date
+ n 1000 session create tunmap
+ wait responses
+ date

contrib/perf/upf-1.bt

@@ -0,0 +1,97 @@
+
+uprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_fd_cb {
+        @pfcp_rx_start_at = nsecs;
+}
+
+uprobe:/usr/local/bin/osmo-upf:up_endpoint_rx_cb {
+	@pfcp_handle_start_at = nsecs;
+}
+
+uprobe:/usr/local/lib/libnftables.*:nft_run_cmd_from_buffer {
+        @nft_run_cmd_from_buffer_start_at = nsecs;
+	@nft_run_cmd_from_buffer_us_val = (uint64)0;
+}
+
+uretprobe:/usr/local/lib/libnftables.*:nft_run_cmd_from_buffer {
+	@nft_run_cmd_from_buffer_us_val = (nsecs - @nft_run_cmd_from_buffer_start_at)/1000;
+	@nft_run_cmd_from_buffer_us = hist(@nft_run_cmd_from_buffer_us_val);
+}
+
+uprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_fd_write_cb {
+        @pfcp_write_cb_start_at = nsecs;
+	@pfcp_write_cb_us_val = (uint64)0;
+}
+uretprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_fd_write_cb {
+	@pfcp_write_cb_us_val = (nsecs - @pfcp_write_cb_start_at) / 1000;
+	@pfcp_write_cb_us = hist(@pfcp_write_cb_us_val);
+}
+
+uprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_endpoint_tx_data_no_logging {
+        @pfcp_tx_start_at = nsecs;
+	@pfcp_tx_us_val = (uint64)0;
+}
+uretprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_endpoint_tx_data_no_logging {
+	@pfcp_tx_us_val = (nsecs - @pfcp_tx_start_at) / 1000;
+	@pfcp_tx_us = hist(@pfcp_tx_us_val);
+}
+
+uretprobe:/usr/local/lib/libosmo-pfcp.*:osmo_pfcp_fd_cb {
+	@pfcp_rx_count = @pfcp_rx_count + 1;
+
+        $pfcp_rx_end_at = nsecs;
+
+
+	if (@pfcp_handle_start_at > @pfcp_rx_start_at) {
+		$pfcp_rx_total_us_val = ($pfcp_rx_end_at - @pfcp_rx_start_at) / 1000;
+
+		$pfcp_decode_us_val = (@pfcp_handle_start_at - @pfcp_rx_start_at) / 1000;
+		@pfcp_decode_us = hist($pfcp_decode_us_val);
+
+		$pfcp_decode_percent_val = 100 * $pfcp_decode_us_val / $pfcp_rx_total_us_val;
+		@pfcp_decode_percent = lhist($pfcp_decode_percent_val, 0, 120, 10);
+
+		$pfcp_handle_us_val = ($pfcp_rx_end_at - @pfcp_handle_start_at) / 1000;
+		@pfcp_handle_us = hist($pfcp_handle_us_val);
+
+		$pfcp_handle_percent_val = 100 * $pfcp_handle_us_val / $pfcp_rx_total_us_val;
+		@pfcp_handle_percent = lhist($pfcp_handle_percent_val, 0, 120, 10);
+
+		if (@nft_run_cmd_from_buffer_start_at > @pfcp_rx_start_at && @nft_run_cmd_from_buffer_us_val) {
+			$nft_run_cmd_from_buffer_percent_val = 100 * @nft_run_cmd_from_buffer_us_val / $pfcp_rx_total_us_val;
+			@nft_run_cmd_from_buffer_percent = lhist($nft_run_cmd_from_buffer_percent_val, 0, 100, 10);
+		}
+
+		if (@pfcp_tx_start_at > @pfcp_rx_start_at && @pfcp_tx_us_val) {
+			$pfcp_tx_percent_val = 100 * @pfcp_tx_us_val / $pfcp_rx_total_us_val;
+			@pfcp_tx_percent = lhist($pfcp_tx_percent_val, 0, 120, 10);
+		}
+		if (@pfcp_write_cb_start_at > @pfcp_rx_start_at && @pfcp_write_cb_us_val) {
+			$pfcp_write_cb_percent_val = 100 * @pfcp_write_cb_us_val / $pfcp_rx_total_us_val;
+			@pfcp_write_cb_percent = lhist($pfcp_write_cb_percent_val, 0, 120, 10);
+		}
+	}
+}
+
+interval:s:1 {
+	$pfcp_rx_per_s_val = @pfcp_rx_count;
+	@pfcp_rx_count = 0;
+	@pfcp_rx_per_s = hist($pfcp_rx_per_s_val);
+}
+
+interval:s:10 {
+        printf("==========================================================");
+        time("%Y-%m-%d %H:%M:%S\n");
+        print(@pfcp_decode_us);
+        print(@pfcp_handle_us);
+	print(@nft_run_cmd_from_buffer_us);
+        print(@pfcp_tx_us);
+        print(@pfcp_write_cb_us);
+
+        print(@pfcp_decode_percent);
+        print(@pfcp_handle_percent);
+	print(@nft_run_cmd_from_buffer_percent);
+	print(@pfcp_tx_percent);
+	print(@pfcp_write_cb_percent);
+
+	print(@pfcp_rx_per_s);
+}

contrib/perf/upf-2.bt

@@ -0,0 +1,46 @@
+BEGIN {
+        @poll__last_nsecs = nsecs;
+}
+
+tracepoint:syscalls:sys_enter_poll /comm == "osmo-upf"/ {
+        $elapsed = (nsecs - @poll__last_nsecs)/1000000;
+        @poll__last_nsecs = nsecs;
+        @poll__ms = hist($elapsed);
+}
+
+uprobe:/usr/local/bin/osmo-upf:up_gtp_action_enable {
+        @up_gtp_action_enable__start = nsecs;
+}
+
+uprobe:/usr/local/lib/libnftables.*:nft_run_cmd_from_buffer {
+        $up_gtp_action_enable__duration = (nsecs - @up_gtp_action_enable__start)/1000;
+        @up_gtp_action_enable__us = hist($up_gtp_action_enable__duration);
+
+        @nft_run_cmd_from_buffer__start = nsecs;
+}
+
+uretprobe:/usr/local/lib/libnftables.*:nft_run_cmd_from_buffer {
+        $nft_run_cmd_from_buffer__duration = (nsecs - @nft_run_cmd_from_buffer__start)/1000000;
+        @nft_run_cmd_from_buffer__ms = hist($nft_run_cmd_from_buffer__duration);
+}
+
+uprobe:/usr/local/lib/libnftables.*:nft_parse_bison_buffer {
+        $up_gtp_action_enable__duration = (nsecs - @up_gtp_action_enable__start)/1000;
+        @up_gtp_action_enable__us = hist($up_gtp_action_enable__duration);
+
+        @nft_run_cmd_from_buffer__start = nsecs;
+}
+
+uretprobe:/usr/local/lib/libnftables.*:nft_parse_bison_buffer {
+        $nft_run_cmd_from_buffer__duration = (nsecs - @nft_run_cmd_from_buffer__start)/1000000;
+        @nft_run_cmd_from_buffer__ms = hist($nft_run_cmd_from_buffer__duration);
+}
+
+interval:s:10 {
+        printf("==========================================================");
+        time("%Y-%m-%d %H:%M:%S\n");
+        print(@poll__ms);
+        print(@logp2__us);
+        print(@up_gtp_action_enable__us);
+        print(@nft_run_cmd_from_buffer__ms);
+}

contrib/perf/upftrace.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+./gen_bpftrace.py \
+"/usr/local/lib/libnftables.*" \
+\
+free_const \
+chain_free \
+set_free \
+flowtable_free \
+flowtable_free \
+obj_free \
+handle_free \
+scope_release \
+cache_free \
+"/usr/lib/x86_64-linux-gnu/libc.*" free \
+\
+ > now.bt
+cat now.bt
+
+sudo bpftrace now.bt

contrib/systemd/osmo-upf.service

@@ -1,11 +1,18 @@
 [Unit]
 Description=Osmocom User Plane Function (UPF)
+After=network-online.target
+Wants=network-online.target
 
 [Service]
 Type=simple
+StateDirectory=osmocom
+WorkingDirectory=%S/osmocom
 Restart=always
+User=osmocom
+Group=osmocom
 ExecStart=/usr/bin/osmo-upf -c /etc/osmocom/osmo-upf.cfg
 RestartSec=2
+AmbientCapabilities=CAP_NET_ADMIN
 
 [Install]
 WantedBy=multi-user.target

debian/changelog

@@ -0,0 +1,23 @@
+osmo-upf (0.1.1) unstable; urgency=medium
+
+  [ Vadim Yanitskiy ]
+  * configure.ac: do not require unused dlopen
+  * Fix missing dash in Redmine project URL
+
+  [ Harald Welte ]
+  * add missing dependencies to libosmo-{gtlv,pfcp} to rpm + dpkg
+
+  [ Neels Hofmeyr ]
+  * add missing COPYING file
+  * debian,RPM,configure: fix packaging (deps etc)
+
+  [ Neels Janosch Hofmeyr ]
+  * configure: set libosmocore >= 1.6.0 like in packaging
+
+ -- Neels Janosch Hofmeyr <nhofmeyr@sysmocom.de>  Wed, 24 Aug 2022 16:50:56 +0200
+
+osmo-upf (0.1.0) testing; urgency=low
+
+  Create new Debian package.
+
+ -- Neels Hofmeyr <nhofmeyr@sysmocom.de>  Tue, 16 Aug 2022 23:17:05 +0200

debian/compat

@@ -1 +1 @@
-9
+10

debian/control

@@ -2,7 +2,7 @@ Source: osmo-upf
 Section: net
 Priority: extra
 Maintainer: Osmocom team <openbsc@lists.osmocom.org>
-Build-Depends: debhelper (>=9),
+Build-Depends: debhelper (>= 10),
                dh-autoreconf,
                autotools-dev,
                autoconf,
@@ -13,11 +13,13 @@ Build-Depends: debhelper (>=9),
                python3-minimal,
                libtalloc-dev,
                libgtpnl-dev (>= 1.2.0),
+               libnftables-dev (>= 1.0.2),
                libosmocore-dev (>= 1.6.0),
+               libosmo-pfcp-dev (>= 0.1.0),
                osmo-gsm-manuals-dev (>= 1.2.0)
 Standards-Version: 3.9.8
-Vcs-Git: git://git.osmocom.org/osmo-upf.git
-Vcs-Browser: https://git.osmocom.org/osmo-upf/
+Vcs-Git: https://gitea.osmocom.org/cellular-infrastructure/osmo-upf
+Vcs-Browser: https://gitea.osmocom.org/cellular-infrastructure/osmo-upf
 Homepage: https://projects.osmocom.org/projects/osmo-upf
 
 Package: osmo-upf

debian/copyright

@@ -1,6 +1,6 @@
 Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: osmo-upf
-Source: git://git.osmocom.org/osmo-upf
+Source: https://gitea.osmocom.org/cellular-infrastructure/osmo-upf
 
 Files:     *
 Copyright: 2021-2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>

debian/osmo-upf.install

@@ -1,4 +1,5 @@
 etc/osmocom/osmo-upf.cfg
 lib/systemd/system/osmo-upf.service
+usr/bin/osmo-pfcp-tool
 usr/bin/osmo-upf
 usr/share/doc/osmo-upf/examples/osmo-upf/osmo-upf.cfg usr/share/doc/osmo-upf/examples

debian/postinst

@@ -0,0 +1,38 @@
+#!/bin/sh -e
+case "$1" in
+	configure)
+		# Create the osmocom group and user (if it doesn't exist yet)
+		if ! getent group osmocom >/dev/null; then
+			groupadd --system osmocom
+		fi
+		if ! getent passwd osmocom >/dev/null; then
+			useradd \
+				--system \
+				--gid osmocom \
+				--home-dir /var/lib/osmocom \
+				--shell /sbin/nologin \
+				--comment "Open Source Mobile Communications" \
+				osmocom
+		fi
+
+		# Fix permissions of previous (root-owned) install (OS#4107)
+		if dpkg --compare-versions "$2" le "0.2.0"; then
+			if [ -e /etc/osmocom/osmo-upf.cfg ]; then
+				chown -v osmocom:osmocom /etc/osmocom/osmo-upf.cfg
+				chmod -v 0660 /etc/osmocom/osmo-upf.cfg
+			fi
+
+			if [ -d /etc/osmocom ]; then
+				chown -v root:osmocom /etc/osmocom
+				chmod -v 2775 /etc/osmocom
+			fi
+
+			mkdir -p /var/lib/osmocom
+			chown -R -v osmocom:osmocom /var/lib/osmocom
+		fi
+		;;
+esac
+
+# dh_installdeb(1) will replace this with shell code automatically
+# generated by other debhelper scripts.
+#DEBHELPER#

doc/Makefile.am

@@ -1,5 +1,4 @@
 SUBDIRS = \
 	examples \
 	manuals \
-	charts \
 	$(NULL)

doc/charts/Makefile.am

@@ -1,24 +0,0 @@
-msc: \
-	$(builddir)/pfcp_msgs.png \
-	$(builddir)/pfcp_msgs_gtp.png \
-	$(NULL)
-
-dot: \
-	$(builddir)/pfcp_overview.png \
-	$(builddir)/pfcp_cp_peer_fsm.png \
-	$(builddir)/pfcp_up_peer_fsm.png \
-	$(builddir)/pfcp_heartbeat_fsm.png \
-	$(builddir)/pfcp_cp_session_fsm.png \
-	$(builddir)/pfcp_up_session_fsm.png \
-	$(builddir)/pfcp_and_gtp.png \
-	$(NULL)
-
-$(builddir)/%.png: $(srcdir)/%.msc
-	mscgen -T png -o $@ $<
-
-$(builddir)/%.png: $(srcdir)/%.dot
-	dot -Tpng $< > $@
-
-.PHONY: poll
-poll:
-	while true; do $(MAKE) msc dot; sleep 1; done

doc/charts/pfcp_and_gtp.dot

@@ -1,20 +0,0 @@
-digraph G {
-rankdir=LR
-labelloc=t; label="PFCP and GTP"
-
-SGSN [label="SGSN\n123.44.0.9"]
-SGWC [label="SGW-C\n123.44.05"]
-subgraph cluster_UPF {
- label="OsmoUPF";
- SGWU [label="SGW-U\n123.44.0.6"];
- GTPk [label="kernel GTP\n123.44.0.6"]
-}
-
-SGSN -> SGWC [label="S4\nGTPv2-C"]
-SGWC -> SGWU [label="Sxa\nPFCP\nSession Establishment:\n"]
-SGSN -> GTPk [label="S4\nGTPv1-U",dir=both]
-
-MS [label="MS\n192.168.104.176"]
-MS -> SGSN [dir=both]
-
-}

doc/charts/pfcp_cp_peer_fsm.dot

@@ -1,39 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP CP peer FSM\nControl Plane side, managing association with remote UP peer"
-
-cp [label="CP function",shape="box"]
-
-cp -> WAIT_ASSOC_SETUP_RESP [label="cp_peer_associate()"]
-
-txrx [label="PFCP socket",shape="box"]
-WAIT_ASSOC_SETUP_RESP -> txrx [label="tx_assoc_setup_req()",style=dotted]
-txrx -> WAIT_ASSOC_SETUP_RESP [label="EV_RX_ASSOC_SETUP_RESP",style=dotted]
-WAIT_ASSOC_SETUP_RESP -> ASSOCIATED [label="Assoc Setup Resp"]
-
-WAIT_ASSOC_SETUP_RESP -> WAIT_ASSOC_SETUP_RESP [label="retry"]
-
-heartbeat [label="PFCP heartbeat FSM",shape=box3d]
-ASSOCIATED -> heartbeat [label="alloc()",style=dotted]
-heartbeat -> ASSOCIATED [label="EV_HEARTBEAT_FAILURE",style=dotted]
-
-txrx2 [label="PFCP socket",shape="box"]
-txrx2 -> ASSOCIATED [label="EV_RX_ASSOC_UPDATE_REQ\n3GPP TS 29.244 6.2.7.3.1",style=dotted]
-GRACEFUL_RELEASE -> txrx2 [label="tx_assoc_update_resp()",style=dotted]
-
-cp_session [label="PFCP CP session FSM",shape=box3d]
-cp -> ASSOCIATED [label="cp_peer_session_create()",style=dotted]
-ASSOCIATED -> cp_session [label="cp_session_create()",style=dotted]
-cp -> cp_session [style=invisible,arrowhead=none]
-
-ASSOCIATED -> GRACEFUL_RELEASE [label="Association Update\nindicating graceful release"]
-
-cp -> ASSOCIATED [label="cp_peer_release()",style=dotted]
-ASSOCIATED -> term [label="cp_peer_release()\nHeartbeat failure"]
-
-ASSOCIATED -> WAIT_ASSOC_SETUP_RESP [label="Heartbeat failure"]
-
-GRACEFUL_RELEASE -> term
-term [shape="octagon"]
-
-}

doc/charts/pfcp_cp_session_fsm.dot

@@ -1,28 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP CP session FSM"
-
-cp [label="CP function",shape=box]
-cp -> WAIT_ESTABLISHMENT_RESP [label="cp_session_create(cp_peer)\niff cp_peer in state ASSOCIATED"]
-
-txrx [label="PFCP socket",shape=box]
-
-WAIT_ESTABLISHMENT_RESP -> txrx [label="tx_session_est_req()",style=dotted]
-txrx -> WAIT_ESTABLISHMENT_RESP [label="EV_RX_SESSION_EST_RESP",style=dotted]
-
-WAIT_ESTABLISHMENT_RESP -> ESTABLISHED [label="Est Resp"]
-
-cp -> ESTABLISHED [label="cp_session_modify()",style=dotted]
-ESTABLISHED -> WAIT_MODIFICATION_RESP [label="cp_session_modify()"]
-WAIT_MODIFICATION_RESP -> txrx [label="tx_session_mod_req()",style=dotted]
-txrx -> WAIT_MODIFICATION_RESP [label="EV_RX_SESSION_MOD_RESP",style=dotted,constraint=false]
-WAIT_MODIFICATION_RESP -> ESTABLISHED [label="Mod Resp"]
-
-cp -> ESTABLISHED [label="cp_session_delete()",style=dotted]
-ESTABLISHED -> WAIT_DELETION_RESP [label="cp_session_delete()"]
-WAIT_DELETION_RESP -> txrx [label="tx_session_del_req()",style=dotted]
-txrx -> WAIT_DELETION_RESP [label="EV_RX_SESSION_DEL_RESP",style=dotted,constraint=false]
-WAIT_DELETION_RESP -> term
-term [shape="octagon"]
-
-}

doc/charts/pfcp_heartbeat_fsm.dot

@@ -1,21 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP heartbeat FSM"
-
-peer [label="PFCP CP/UP peer FSM",shape=box3d]
-txrx [label="PFCP socket",shape=box]
-
-peer -> IDLE [label="alloc()"]
-IDLE -> WAIT_HEARTBEAT_RESP -> IDLE
-WAIT_HEARTBEAT_RESP -> term
-term [shape="octagon"]
-
-WAIT_HEARTBEAT_RESP -> txrx [label="tx_heartbeat_req()",style=dotted]
-txrx -> WAIT_HEARTBEAT_RESP [label="HEARTBEAT_EV_RX_RESP",style=dotted]
-
-term -> peer [label="PEER_EV_HEARTBEAT_FAILURE",style=dotted]
-
-txrx2 [label="PFCP socket",shape=box]
-txrx2 -> txrx2 [label="rx Heartbeat Req\ntx Heartbeat Resp",style=dotted]
-
-}

doc/charts/pfcp_overview.dot

@@ -1,23 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP Overview\n3GPP TS 29.244 3.1, 5.8.1"
-
-subgraph cluster_N1_CP {
-	label="Node: Control Plane function";style=dotted
-	N1_E_CP [label="CP Entity"]
-}
-
-subgraph cluster_N2_UP {
-	label="Node: User Plane function\nNode ID: my-userplane.com\n(FQDN may provide multiple PFCP Entities)";style=dotted
-	N2_E_UP [label="UP Entity\n8.7.6.1"]
-	N2_E_UP2 [label="UP Entity\n8.7.6.2"]
-}
-
-subgraph cluster_N3_UP {
-	label="Node: User Plane function\nNode ID: 1.2.3.4\n(IP address means only one PFCP Entity)";style=dotted
-	N3_E_UP [label="UP Entity\n1.2.3.4\n(osmo-upf)"]
-}
-
-N1_E_CP -> N3_E_UP [label="PFCP Request"]
-N1_E_CP -> N2_E_UP
-}

doc/charts/pfcp_up_peer_fsm.dot

@@ -1,27 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP UP peer FSM\nUser Plane side, managing association with remote CP peer"
-
-txrx [label="PFCP socket",shape="box"]
-
-txrx -> NOT_ASSOCIATED [label="rx PFCP msg from\nnew remote IP"]
-txrx -> NOT_ASSOCIATED [label="EV_RX_ASSOC_SETUP_REQ",style=dotted]
-
-NOT_ASSOCIATED -> ASSOCIATED [label="Assoc Setup Req",shape="box"]
-
-heartbeat [label="PFCP heartbeat FSM",shape=box3d]
-ASSOCIATED -> heartbeat [label="alloc()",style=dotted]
-heartbeat -> ASSOCIATED [label="EV_HEARTBEAT_FAILURE",style=dotted]
-
-txrx -> ASSOCIATED [label="EV_RX_SESSION_EST_REQ",style=dotted]
-up_session [label="PFCP UP session FSM",shape=box3d]
-ASSOCIATED -> up_session [label="up_session_create()",style=dotted]
-
-txrx -> ASSOCIATED [label="EV_RX_ASSOC_UPD_REQ",style=dotted]
-ASSOCIATED -> GRACEFUL_RELEASE [label="Association Update\nindicating graceful release"]
-
-ASSOCIATED -> term [label="Heartbeat failure"]
-GRACEFUL_RELEASE -> term
-term [shape="octagon"]
-
-}

doc/charts/pfcp_up_session_fsm.dot

@@ -1,21 +0,0 @@
-digraph G {
-rankdir=TB
-labelloc=t; label="PFCP UP session FSM"
-
-peer [label="PFCP UP peer FSM",shape=box3d]
-peer -> ESTABLISHED [label="rx_session_est_req()"]
-
-txrx [label="PFCP socket",shape="box"]
-txrx2 [label="PFCP socket",shape="box"]
-
-txrx -> ESTABLISHED [label="EV_RX_SESSION_MOD_REQ",style=dotted]
-ESTABLISHED -> txrx [label="tx_session_mod_resp()",style=dotted,constraint=false]
-ESTABLISHED -> ESTABLISHED [label="Mod"]
-
-txrx2 -> ESTABLISHED [label="EV_RX_SESSION_DEL_REQ",style=dotted]
-ESTABLISHED -> txrx2 [label="tx_session_del_resp()",style=dotted,constraint=false]
-
-ESTABLISHED -> term [label="Deletion"]
-term [shape="octagon"]
-
-}

doc/examples/osmo-upf/osmo-upf-create-dev.cfg

@@ -1,16 +1,15 @@
 log stderr
  logging filter all 1
  logging color 1
- logging print level 1
- logging print category 1
  logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
  logging print file basename last
- logging print extended-timestamp 1
- logging level set-all info
-#logging level set-all debug
+ logging print level 1
+ logging level set-all notice
 
 timer pfcp x24 5000
 pfcp
  local-addr 127.0.0.1
-gtp
+tunend
  dev create apn23

doc/examples/osmo-upf/osmo-upf-mockup.cfg

@@ -1,19 +1,17 @@
 log stderr
  logging filter all 1
  logging color 1
- logging print level 1
- logging print category 1
  logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
  logging print file basename last
- logging print extended-timestamp 1
- logging level set-all debug
+ logging print level 1
  logging level set-all notice
- logging level set-all info
 
 timer pfcp x24 5000
 pfcp
  local-addr 127.0.0.1
-gtp
+tunend
  mockup
-nft
+tunmap
  mockup

doc/examples/osmo-upf/osmo-upf.cfg

@@ -1,14 +1,12 @@
 log stderr
  logging filter all 1
  logging color 1
- logging print level 1
- logging print category 1
  logging print category-hex 0
+ logging print category 1
+ logging timestamp 0
  logging print file basename last
- logging print extended-timestamp 1
- logging level set-all debug
+ logging print level 1
  logging level set-all notice
- logging level set-all info
 
 timer pfcp x24 5000
 pfcp

doc/manuals/chapters/netinst.adoc

@@ -0,0 +1,57 @@
+[[netinst]]
+== Local GTP Addresses / Network Instance
+
+PFCP features optional Network Instance IEs, in which the CPF may tell the UPF which local network interface to use for
+a PDR and/or a FAR.
+
+NOTE:: osmo-upf only evaluates the Network Instances configured in PDRs. Since osmo-upf always pairs a PDR+FAR with
+another PDR+FAR in reverse direction, each side's PDR is sufficient.
+
+Network Instance IEs affect both the tunend and the tunmap use cases, as well as which local IP address is returned
+in the PFCP response
+
+1. Look up Network Instance name in the osmo-upf.cfg `netinst` section, to obtain a local IP address.
+2. Depending on use case:
+  - tunend: create the tunnel on a GTP device matching the local IP address, see <<gtp_module>>.
+  - tunmap: use the local IP address in the netfilter ruleset, see <<nftables>>.
+3. Usually, return the chosen local IP address in the F-TEID IE of the Created PDR IE in the PFCP response.
+
+Network Instance configuration consists of {name, IP address} pairs.
+
+NOTE:: As soon as a `netinst` configuration is nonempty, receiving an undefined Network Instance name results in a PFCP
+Reject response, and a log message on cateogry `session`, level `NOTICE`. To make the PFCP return success, add the
+failing name to the `netinst` config.
+
+=== netinst for tunend
+
+The following configuration sets up two GTP devices for tunend, expecting Network Instance names `access1` or `access2`:
+
+----
+tunend
+ dev create apn1 10.0.0.1
+ dev create apn2 10.0.0.2
+netinst
+ add access1 10.0.0.1
+ add access2 10.0.0.2
+----
+
+For example, if a Create PDR IE indicates Network Instance = `access1`, a GTP tunnel is set up in GTP kernel device
+`apn1`. For `access2`, use `apn2`.
+
+=== netinst for tunmap
+
+For the tunmap use case, it is sufficient to configure `netinst` entries, without any addition to the `tunmap` section.
+The following example configures various interfaces for tunmap, to match Network Instance names received in PFCP:
+
+----
+tunmap
+ table-name osmo-upf
+netinst
+ add access1 10.0.0.1
+ add access2 10.0.0.2
+ add core1 9.0.0.1
+ add core2 9.0.0.2
+----
+
+For example, a Create PDR indicating a Network Instance of `core1` will result in an nftables rule that receives packets
+on local address `9.0.0.1`.

doc/manuals/chapters/overview.adoc

@@ -22,3 +22,189 @@ The aim is to provide:
 - 1000 modifications of tunnel state per second (add/remove/modify),
 - 4-8 Gbps throughput,
 - 100-125k concurrent GTP tunnels.
+
+A typical network scenario using OsmoUPF is illustrated in the following
+diagram:
+
+.Typical network architecture used with OsmoUPF
+[graphviz]
+----
+digraph G {
+  rankdir = LR;
+
+  UE [label="UE\n(3G phone)"]
+
+  subgraph cluster_hnbgw_mgw_upf {
+    style=dotted
+    HNBGW -> UPF [label="PFCP",constraint=false]
+    UPF [label=OsmoUPF,style=bold]
+  }
+
+  subgraph cluster_hnbgw_mgw_upf2 {
+    style=dotted
+    SGSN -> UPF2 [label="PFCP",constraint=false]
+    UPF2 [label=OsmoUPF,style=bold]
+  }
+
+  subgraph cluster_hnbgw_mgw_upf3 {
+    style=dotted
+    GGSN -> UPF3 [label="PFCP",constraint=false]
+    UPF3 [label=OsmoUPF,style=bold]
+  }
+
+  hNodeB [shape="box",label="hNodeB\n(3G femto cell)"]
+
+  UE -> hNodeB [label="Uu"]
+  hNodeB -> HNBGW [label="Iuh",style=dashed]
+  STP [label="STP\n(SCCP/M3UA)"]
+  HNBGW -> STP -> SGSN [label="IuPS",style=dashed]
+  SGSN -> GGSN [label="GTP-C",style="dashed"]
+  hNodeB -> UPF -> UPF2 -> UPF3 [label="GTP-U"]
+  UPF3 -> internet [label="apn"]
+}
+----
+
+NOTE: at the time of writing this section, the only Osmocom component providing
+a PFCP CPF interface is OsmoHNBGW. PFCP support has not yet made its way into
+OsmoSGSN nor OsmoGGSN.
+
+=== the PFCP interface
+
+PFCP is specified by 3GPP TS 29.244.
+
+OsmoUPF implements a PFCP User Plane Function interface, listening for PFCP
+requests from PFCP Control Plane Function clients, to carry out proxy-relaying
+and encapsulation/decapsulation of GTP tunnels.
+
+OsmoUPF does not support the complete PFCP feature set. It detects exactly two
+use cases that will provide service of actual GTP tunnels:
+
+.tunend use case
+----
+Access                 osmo-upf              Core
+ PGW                      |              PDN/internet
+  |                PDR1:  > FAR1:             |
+  |                IP/GTP | IP                |
+  |        ------> F-TEID |            -----> |
+  |                       |                   |
+  |                FAR2:  < PDR2:             |
+  |                IP/GTP | IP                |
+  | F-TEID <------        | UE IP addr <----- |
+----
+
+* `tunend`: GTP tunnel encapsulation/decapsulation:
+  - One Packet Detection Rule (PDR) accepts a GTP tunnel from the Access side
+    with an Outer Header Removal.
+  - This PDR uses a Forwarding Action Rule (FAR) for plain IP towards Core.
+  - Another PDR accepts plain IP on a specific IP address from Core.
+  - The second PDR uses a FAR towards Access with Outer Header Creation for GTP.
+
+.tunmap use case
+----
+Access                 osmo-upf                 Core
+ PGW                      |                     PGW
+  |                PDR1:  > FAR1:                |
+  |                IP/GTP | IP/GTP               |
+  |        ------> F-TEID |        -----> F-TEID |
+  |                       |                      |
+  |                FAR2:  < PDR2:                |
+  |                IP/GTP | IP/GTP               |
+  | F-TEID <------        | F-TEID <-----        |
+----
+
+* `tunmap`: GTP tunnel forwarding:
+  - One Packet Detection Rule (PDR) accepts a GTP tunnel from the Access side
+    with an Outer Header Removal.
+  - This PDR uses a Forwarding Action Rule (FAR) towards Core with an Outer
+    Header Creation for GTP.
+  - A second PDR+FAR pair like above, with Access and Core swapped.
+
+Access and Core must be indicated by the Source Interface IE (PDR) and
+Destination Interface IE (FAR) in PFCP.
+
+Any set of rules only partially or not at all matching the above PDR and FAR
+rules will not result in any actions on the GTP user plane, but will still
+return a successful outcome in the PFCP messages.
+
+For example, a rule set using a Source Interface other than "Access" or "Core" results
+in a PFCP no-op, returning PFCP responses with successful outcome, but not
+providing any GTP-U service.
+
+This is a direct result of:
+
+- allowing PFCP rule sets to be setup incrementally by several subsequent PFCP
+  messages, and of
+- OsmoUPF using Linux kernel features for the GTP user plane, where there is
+  either a full bidirectional GTP tunnel in place or none at all.
+
+For example, for `tunmap`, a typical CPF will establish a PFCP session in two
+steps: first request a local F-TEID from the UPF before passing on a data
+service request from Access to Core. When the Core side has responded with its
+GTP details, the PFCP session at the UPF is updated (Session Modifification),
+to form a complete PFCP rule set.
+
+.Typical sequence of establishing a GTP-U tunnel relay
+["mscgen"]
+----
+msc {
+	hscale="1";
+	sgsn[label="SGSN"],sgwc[label="SGW-C"],sgwu[label="SGW-U"],pgwc[label="PGW-C"];
+
+	sgsn << pgwc [label="Access"];
+	sgsn >> pgwc [label="Core"];
+
+	sgsn => sgwc [label="GTP Create Session Request\n\n\n"];
+
+	|||;
+
+	sgwc => sgwu [label="PFCP Session Establishment Request\n\n2x Create PDR\nF-TEID = CHOOSE"];
+
+	|||;
+
+	sgwc <= sgwu [label="PFCP Session Establishment Response\n\n2x Created PDR\nwith chosen local F-TEID"];
+
+	|||;
+
+	sgwc => pgwc [label="GTP Create Session Request\nwith chosen local F-TEID towards Core"];
+	sgwc <= pgwc [label="GTP Create Session Response\nwith remote F-TEID at Core"];
+
+	|||;
+
+	sgwc => sgwu [label="PFCP Session Modification Request\n\nUpdate FAR\nwith remote F-TEID at Core"];
+
+	|||;
+
+	sgwc <= sgwu [label="PFCP Session Modification Response\n\n\n"];
+
+	|||;
+
+	sgsn <= sgwc [label="GTP Create Session Response\n\n\n"];
+}
+----
+
+The OsmoUPF logging as well as the VTY interface yield information on whether a
+ruleset results in an actual bidirectional GTP tunnel being set up.
+
+
+=== the GTP interface
+
+OsmoUPF requires the following Linux kernel features to provide the GTP user
+plane functionality:
+
+- the Linux kernel GTP module for encapsulation/decapsulation between GTP and
+  plain IP.
+- the Linux netfilter nftables feature for relaying GTP, i.e. forwarding between
+  two GTP tunnels.
+
+Tunnel relaying with netfilter requires at least Linux kernel 5.17.
+
+To be able to interact with these Linux kernel features, the osmo-upf binary
+needs cap_net_admin privileges, as in:
+
+----
+sudo setcap cap_net_admin+pe /usr/bin/osmo-upf
+----
+
+Without above Linux kernel features, or when no cap_net_admin is available,
+OsmoUPF is only useful for testing PFCP clients: the GTP features may be run in
+mockup mode, so that OsmoUPF serves as a "dry run" PFCP server.

doc/manuals/chapters/running.adoc

@@ -0,0 +1,269 @@
+== Running OsmoUPF
+
+The OsmoUPF executable (`osmo-upf`) offers the following command-line
+arguments:
+
+=== SYNOPSIS
+
+*osmo-upf* [-h|-V] [-D] [-c 'CONFIGFILE']
+
+=== OPTIONS
+
+*-h, --help*::
+	Print a short help message about the supported options
+*-V, --version*::
+	Print the compile-time version number of the OsmoHNBGW program
+*-D, --daemonize*::
+	Fork the process as a daemon into background.
+*-c, --config-file 'CONFIGFILE'*::
+	Specify the file and path name of the configuration file to be
+	used. If none is specified, use `osmo-upf.cfg` in the current
+	working directory.
+
+=== Multiple instances
+
+Running multiple instances of `osmo-upf` on the same computer is possible if
+all interfaces (VTY, CTRL, PFCP) are separated using the appropriate
+configuration options. The IP based interfaces are binding to local host by
+default. In order to separate the processes, the user has to bind those
+services to different ports, or different specific IP addresses.
+
+The VTY and the Control interface can be bound to IP addresses from the loopback
+address range, for example:
+
+----
+line vty
+ bind 127.0.0.2
+ctrl
+ bind 127.0.0.2
+----
+
+The PFCP port is specified to be fixed as port 8805. Hence, each osmo-upf
+process needs to run on a distinct local interface:
+
+----
+pfcp
+  local-addr 10.9.0.2
+----
+
+For GTP encapsulation/decapsulation and GTP tunnel relaying, osmo-upf depends on
+the IP addresses configured at the Linux kernel GTP module, and the IP addresses
+negotiated within PFCP by the control plane function.
+
+If multiple `osmo-upf` processes are running on the same Linux kernel, each
+`osmo-upf` needs to be configured with a distinct netfilter table name, so that
+naming of individual tunnel rulesets does not collide:
+
+----
+tunmap
+ table-name osmo-upf-2
+----
+
+=== Configure PFCP Server
+
+The following example configures OsmoUPF to listen for PFCP association requests
+from Control Plane Function entities on local interface 10.9.8.7, port 8805:
+
+----
+pfcp
+ local-addr 10.9.8.7
+----
+
+3GPP TS 29.244 4.2.2 specifies that PFCP Request messages shall be sent to UDP
+port 8805, i.e. the PFCP port is fixed as 8805 and currently not configurable in
+osmo-upf.
+
+Setting a 'local-addr' is required: the PFCP protocol features a Node ID, which
+uniquely identifies PFCP peers across different interfaces. According to the
+PFCP specification, the Node ID can be a fully-qualified domain name (FQDN) or
+an IP address. Currently, osmo-upf has no support for using an FQDN as Node
+ID, and so far uses the 'local-addr' as local Node ID -- hence the 'local-addr'
+must not be "0.0.0.0", which is an unfortunate consequence. This is likely to
+improve in the future, see https://osmocom.org/issues/5682 .
+
+=== Linux Kernel Features
+
+OsmoUPF uses two distinct Linux kernel features:
+
+* The GTP module is used for `tunend`: GTP encapsulation/decapsulation from/to
+  "the internet".
+
+* The netfilter framework and nftables are used for `tunmap`: GTP tunnel proxying,
+  also known as tunnel forwarding or tunnel mapping.
+
+.Linux kernel feature usage
+[graphviz]
+----
+include::upf_gtp_roles.dot[]
+----
+
+GTP kernel module configuration in the `tunend` section can be omitted for sites
+that serve only as GTP forwarding proxy, without encapsulation/decapsulation of
+GTP payloads -- except to provide GTP Echo service, see <<gtp_echo>>.
+
+Netfilter configuration in the `tunmap` section can be omitted for sites only
+serving as GTP tunnel endpoint.
+
+[[gtp_module]]
+=== Configure Linux Kernel GTP Module for `tunend`
+
+The Linux kernel GTP module is used for the `tunend` use case, i.e. GTP
+encapsulation/decapsulation from/to "the internet".
+
+To use the GTP kernel module, OsmoUPF requires a GTP device, which is a
+dedicated network device provided by the Linux kernel, serving as GTP tunnel
+endpoint. It is typically named like "apn0".
+
+`osmo-upf` can either create a GTP device on startup, or use a pre-existing GTP
+device. To en/decapsulate GTP, the APN device needs to be assigned an IP address
+range that matches the UE IP addresses that are configured in GTP-C / PFCP.
+
+The following configuration placed in `osmo-upf.cfg` creates a GTP device called
+`apn23` on startup of osmo-upf, which is destroyed on program exit. It listens
+for GTP on local IP address `1.2.3.4`:
+
+----
+tunend
+ dev create apn23 1.2.3.4
+----
+
+TODO:: `osmo-upf` is not yet able to configure this network device's IP address
+range, MTU etc.
+
+The following configuration placed in `osmo-upf.cfg` uses a pre-existing device
+called `apn42`:
+
+----
+tunend
+ dev use apn42 2.3.4.5
+----
+
+GTP kernel devices can be managed manually using the `gtp-link` program
+available from the 'libgtpnl' project:
+
+----
+# gtp-link add apn42
+(keep this process running)
+# ip addr add dev apn42 192.168.42.1/24
+
+$ osmo-upf -c osmo-upf.cfg
+----
+
+It is possible to configure multiple GTP devices in `osmo-upf.cfg`. Depending on
+the Network Instance name, osmo-upf creates tunnel endpoints on the GTP device
+with a matching IP address:
+
+- The Network Instance IE in the PDR on the Access side determines the local IP
+  address to use, see <<netinst>>.
+- This local IP address in turn determines the GTP device to use.
+
+It is possible for a GTP device to listen on ANY -- just omit the IP address in
+the `dev` config. In this case, all Network Instance names will be served by
+this GTP device. When using ANY, there should be exactly one GTP dev configured.
+
+[[nftables]]
+=== Configure Linux netfilter for `tunmap`
+
+The Linux kernel netfilter module is used for GTP tunnel proxying, also known as
+tunnel forwarding or tunnel mapping.
+
+When using the netfilter module, you may set up `osmo-upf.cfg` for:
+- GTP Echo (required)
+- nft table name (optional)
+
+[[gtp_echo]]
+==== GTP Echo
+
+You need to ensure that OsmoUPF responds to GTP Echo requests.
+- A GTP device configured for `tunend` implicitly includes a GTP Echo service.
+- For `tunmap`, no GTP Echo mechanism is implemented.
+
+So, when your use case is `tunmap` only, you should still add a GTP device as
+for `tunend`, only to provide the GTP Echo service.
+
+Here are some options to do so:
+
+If you have no GTP devices configured in `osmo-upf.cfg` yet, you can add a
+single GTP device without a specific IP address, in order to respond to GTP-U
+Echo requests on all interfaces to anyone that is asking:
+
+----
+tunend
+ dev create gtp-echo
+----
+
+Note that `gtp-echo` is just an arbitrary GTP device name, choose any string
+that makes a valid network device name and is still available, as in the `dev`
+argument in the `ip addr show dev` command on Linux.
+
+This will bind osmo-upf on 0.0.0.0:2152 to respond to GTP Echo requests.
+
+If you would like to limit GTP Echo responses to specific network interfaces,
+you need to add a separate GTP device per local IP address:
+
+----
+tunend
+ dev create gtp-echo1 192.168.0.23
+ dev create gtp-echo2 10.9.8.17
+----
+
+This will bind osmo-upf only on 192.168.0.23:2152 and 10.9.8.17:2152 to respond
+to GTP Echo requests.
+
+For creating and manipulating a GTP device in more versatile ways, see
+<<gtp_module>>.
+
+==== nft Table Name
+
+For `tunmap`, `osmo-upf` creates a new nft table, under which it submits
+rule sets for GTP tunnel proxying. This table name defaults to `osmo-upf`. A
+custom table name can be configured in `osmo-upf.cfg` like this:
+
+----
+tunmap
+ table-name my-table-name
+----
+
+When running more than one osmo-upf process on a system, pick distinct table
+names to avoid name collisions in the nftables rulesets.
+
+=== IP Forwarding
+
+In order to allow forwarding GTP payloads, the Linux operating system must
+be configured to allow IP forwarding.
+
+Note that there are many distribution-specific ways to configure this, and there
+might be higher-level firewall rule management software available like `ufw`.
+You should configure firewall rules matching your distribution and setup.
+
+To allow IP forwarding from and to all interfaces globally in a reboot-safe way,
+you may put a line like this in /etc/sysctl.conf:
+
+----
+net.ipv4.ip_forward=1
+----
+
+To do the same in an ad-hoc way that is not reboot safe but takes effect
+immediately:
+
+----
+sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
+----
+
+It is also possible to instruct the firewall to allow IP forwarding for specific
+network devices only. For example, on a Debian based system, place an nft
+ruleset like this in `/etc/nftables.conf`:
+
+----
+define gtp_netdevs = { eth0, eth23 };
+
+table inet filter {
+      chain forward {
+              type filter hook forward priority filter; policy drop;
+              iifname $gtp_netdevs oifname $gtp_netdevs udp dport 2152 accept
+      }
+}
+----
+
+This ruleset allows IP forwarding, but limited to the GTP-U port 2152,
+and to two specific network devices eth0 and eth23.

doc/manuals/chapters/upf_gtp_roles.dot

@@ -0,0 +1,31 @@
+digraph G {
+rankdir=LR
+sgsn [label="SGSN"]
+
+subgraph cluster_sgw {
+	style=invisible
+	sgwc [label="SGW-C"]
+	sgwu [label="OsmoUPF as SGW-U\ntunnel proxy\n*netfilter* kernel module",style=bold,shape=box]
+	sgwc -> sgwu [label="PFCP",constraint=false]
+}
+
+subgraph cluster_pgw {
+	style=invisible
+	pgwc [label="PGW-C"]
+	pgwu [label="OsmoUPF as PGW-U\ntunnel proxy\n*netfilter* kernel module",style=bold,shape=box]
+	pgwc -> pgwu [label="PFCP",constraint=false]
+}
+
+subgraph cluster_tdf {
+	style=invisible
+	tdfc [label="TDF-C"]
+	tdfu [label="OsmoUPF as TDF-U\ntunnel en-/decaps\n*GTP* kernel module",style=bold,shape=box]
+	tdfc -> tdfu [label="PFCP",constraint=false]
+}
+
+pdn [label="PDN\n'the internet'"]
+
+sgsn -> sgwc -> pgwc -> tdfc [label="GTP-C"]
+sgsn -> sgwu -> pgwu -> tdfu [label="GTP-U",dir=both]
+tdfu -> pdn [label="IP",dir=both]
+}

doc/manuals/osmoupf-usermanual.adoc

@@ -9,6 +9,10 @@ include::./common/chapters/preface.adoc[]
 
 include::{srcdir}/chapters/overview.adoc[]
 
+include::{srcdir}/chapters/running.adoc[]
+
+include::{srcdir}/chapters/netinst.adoc[]
+
 include::./common/chapters/vty.adoc[]
 
 include::./common/chapters/logging.adoc[]

include/osmocom/Makefile.am

@@ -1,3 +1,4 @@
 SUBDIRS = \
 	upf \
+	pfcptool \
 	$(NULL)

include/osmocom/pfcptool/Makefile.am

@@ -0,0 +1,4 @@
+noinst_HEADERS = \
+	gtp_probe.h \
+	pfcp_tool.h \
+	$(NULL)

include/osmocom/pfcptool/gtp_probe.h

@@ -0,0 +1,28 @@
+#pragma once
+
+#include <osmocom/core/sockaddr_str.h>
+#include <osmocom/core/select.h>
+#include <osmocom/core/socket.h>
+
+struct pfcp_tool_gtp_tun;
+
+struct gtp_probe_target {
+	struct osmo_sockaddr_str cfg;
+
+	struct osmo_sockaddr addr;
+	struct osmo_fd ofd;
+};
+
+struct gtp_probe_state {
+	struct gtp_probe_target ip_target;
+	struct gtp_probe_target gtp_target;
+};
+
+int gtp_probe_target_bind(struct gtp_probe_state *s, struct gtp_probe_target *t);
+
+int gtp_probe_bind_ip_target(struct gtp_probe_state *state);
+int gtp_probe_bind_gtp_target(struct gtp_probe_state *state);
+
+int gtp_probe_run_tunend(struct gtp_probe_state *s, struct pfcp_tool_gtp_tun *inject_gtp,
+			 struct osmo_sockaddr_str *ue_ip,
+			 struct gtp_probe_target *expect_ip);

include/osmocom/pfcptool/pfcp_tool.h

@@ -32,6 +32,7 @@
 #include <osmocom/pfcp/pfcp_msg.h>
 
 #include <osmocom/upf/up_gtp_action.h>
+#include <osmocom/pfcptool/gtp_probe.h>
 
 struct osmo_tdef;
 struct ctrl_handle;
@@ -51,30 +52,44 @@ struct pfcp_tool_peer {
 	struct llist_head sessions;
 };
 
-struct pfcp_tool_teid_pair {
-	uint32_t local;
-	uint32_t remote;
+struct pfcp_tool_gtp_tun_ep {
+	struct osmo_sockaddr_str addr;
+	uint32_t teid;
+};
+
+struct pfcp_tool_gtp_tun {
+	uint16_t local_pdr_id;
+	struct pfcp_tool_gtp_tun_ep local;
+	struct pfcp_tool_gtp_tun_ep remote;
+};
+
+struct pfcp_tool_tunend {
+	struct pfcp_tool_gtp_tun access;
+	struct {
+		struct osmo_sockaddr_str ue_local_addr;
+	} core;
+};
+
+struct pfcp_tool_tunmap {
+	struct pfcp_tool_gtp_tun access;
+	struct pfcp_tool_gtp_tun core;
 };
 
 struct pfcp_tool_session {
 	struct llist_head entry;
 
-	enum up_gtp_action_kind gtp_action;
-
 	struct pfcp_tool_peer *peer;
 	uint64_t cp_seid;
 	struct osmo_pfcp_ie_f_seid up_f_seid;
 
-	struct {
-		struct pfcp_tool_teid_pair teid;
-		struct osmo_sockaddr_str gtp_ip;
-	} access;
+	enum up_gtp_action_kind kind;
+	union {
+		/* En-/De-capsulate GTP: add/remove a GTP header and forward the GTP payload from/to plain IP. */
+		struct pfcp_tool_tunend tunend;
 
-	struct {
-		struct pfcp_tool_teid_pair teid;
-		struct osmo_sockaddr_str gtp_ip;
-		struct osmo_sockaddr_str ue_addr;
-	} core;
+		/* Tunnel-map GTP: translate from one TEID to another and forward */
+		struct pfcp_tool_tunmap tunmap;
+	};
 };
 
 struct g_pfcp_tool {
@@ -87,6 +102,11 @@ struct g_pfcp_tool {
 
 	struct osmo_pfcp_endpoint *ep;
 	struct llist_head peers;
+
+	uint32_t next_teid_state;
+	uint32_t next_ue_addr_state;
+
+	struct gtp_probe_state gtp_probe;
 };
 
 extern struct g_pfcp_tool *g_pfcp_tool;
@@ -95,7 +115,7 @@ void g_pfcp_tool_alloc(void *ctx);
 void pfcp_tool_vty_init_cfg();
 void pfcp_tool_vty_init_cmds();
 
-int pfcp_tool_mainloop();
+int pfcp_tool_mainloop(int poll);
 
 struct pfcp_tool_peer *pfcp_tool_peer_find_or_create(const struct osmo_sockaddr *remote_addr);
 struct pfcp_tool_session *pfcp_tool_session_find_or_create(struct pfcp_tool_peer *peer, uint64_t cp_seid,
@@ -104,3 +124,6 @@ void pfcp_tool_rx_msg(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_msg *m, st
 
 int peer_tx(struct pfcp_tool_peer *peer, struct osmo_pfcp_msg *m);
 uint64_t peer_new_seid(struct pfcp_tool_peer *peer);
+
+uint32_t pfcp_tool_new_teid(void);
+int pfcp_tool_next_ue_addr(struct osmo_sockaddr_str *dst);

include/osmocom/upf/Makefile.am

@@ -1,4 +1,5 @@
 noinst_HEADERS = \
+	netinst.h \
 	up_endpoint.h \
 	up_peer.h \
 	up_session.h \
@@ -6,5 +7,6 @@ noinst_HEADERS = \
 	upf_gtp.h \
 	upf_gtpu_echo.h \
 	upf_nft.h \
+	upf_tun.h \
 	up_gtp_action.h \
 	$(NULL)

include/osmocom/upf/netinst.h

@@ -0,0 +1,44 @@
+/*
+ * (C) 2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ * All Rights Reserved.
+ *
+ * Author: Neels Janosch Hofmeyr <nhofmeyr@sysmocom.de>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#pragma once
+
+#include <osmocom/core/linuxlist.h>
+#include <osmocom/core/sockaddr_str.h>
+
+struct vty;
+
+struct network_instance {
+	struct llist_head entry;
+
+	char *name;
+	struct osmo_sockaddr_str addr;
+};
+
+const struct network_instance *netinst_add(void *ctx, struct llist_head *list, const char *name, const char *addr,
+					   const char **errmsg);
+const struct network_instance *netinst_find(struct llist_head *list, const char *name);
+const struct network_instance *netinst_first(struct llist_head *list);
+int netinst_clear(struct llist_head *list);
+
+int netinst_vty_write(struct vty *vty, struct llist_head *list, const char *indent, const char *name_or_null);

include/osmocom/upf/up_endpoint.h

@@ -35,14 +35,14 @@ struct osmo_sockaddr;
 struct up_endpoint {
 	struct osmo_pfcp_endpoint *pfcp_ep;
 
+	/* list of struct up_peer. */
 	struct llist_head peers;
 
-	uint64_t next_seid_state;
-	uint32_t next_teid_state;
+	uint64_t next_up_seid_state;
 };
 
-struct up_endpoint *up_endpoint_init(void *ctx, const struct osmo_sockaddr *local_addr);
+struct up_endpoint *up_endpoint_alloc(void *ctx, const struct osmo_sockaddr *local_addr);
+int up_endpoint_bind(struct up_endpoint *up_ep);
 void up_endpoint_free(struct up_endpoint **ep);
 
-uint64_t up_endpoint_next_seid(struct up_endpoint *ep);
-uint32_t up_endpoint_next_teid(struct up_endpoint *ep);
+uint64_t up_endpoint_next_up_seid(struct up_endpoint *ep);

include/osmocom/upf/up_gtp_action.h

@@ -40,7 +40,7 @@ struct up_session;
 
 enum up_gtp_action_kind {
 	UP_GTP_DROP,
-	UP_GTP_U_ENDECAPS,
+	UP_GTP_U_TUNEND,
 	UP_GTP_U_TUNMAP,
 };
 
@@ -48,16 +48,16 @@ struct up_gtp_action {
 	struct llist_head entry;
 	struct up_session *session;
 
-	uint16_t pdr_core;
 	uint16_t pdr_access;
+	uint16_t pdr_core;
 
 	enum up_gtp_action_kind kind;
 	union {
 		/* En-/De-capsulate GTP: add/remove a GTP header and forward the GTP payload from/to plain IP. */
-		struct upf_gtp_tun_desc endecaps;
+		struct upf_tunend tunend;
 
 		/* Tunnel-map GTP: translate from one TEID to another and forward */
-		struct upf_nft_tunmap_desc tunmap;
+		struct upf_tunmap tunmap;
 	};
 
 	/* volatile loop variable to match up wanted and actually present GTP actions */

include/osmocom/upf/up_peer.h

@@ -44,6 +44,7 @@ struct up_peer {
 	struct llist_head entry;
 
 	struct osmo_fsm_inst *fi;
+	/* backpointer */
 	struct up_endpoint *up_endpoint;
 
 	/* peer's remote address */

include/osmocom/upf/up_session.h

@@ -41,17 +41,12 @@ enum up_session_fsm_event {
 	UP_SESSION_EV_USE_COUNT_ZERO,
 };
 
-enum up_session_kind {
-	UP_SESSION_DROP,
-	UP_SESSION_GTP_U_ENDECAPS,
-	UP_SESSION_GTP_U_FORW,
-};
-
 struct up_session {
 	struct hlist_node node_by_up_seid;
 	struct hlist_node node_by_cp_seid;
 
 	struct osmo_fsm_inst *fi;
+	/* backpointer */
 	struct up_peer *up_peer;
 
 	struct osmo_pfcp_ie_f_seid cp_f_seid;
@@ -60,15 +55,18 @@ struct up_session {
 	struct osmo_use_count use_count;
 	struct osmo_use_count_entry use_count_buf[8];
 
+	/* llist of struct pdr */
 	struct llist_head pdrs;
+	/* llist of struct far */
 	struct llist_head fars;
+	/* llist of struct chosen_f_teid */
 	struct llist_head chosen_f_teids;
 
+	/* llist of struct up_gtp_action */
 	struct llist_head active_gtp_actions;
 };
 
-struct up_session *up_session_find_or_add(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid,
-					  const struct osmo_pfcp_ie_f_seid *up_f_seid);
+struct up_session *up_session_find_or_add(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid);
 struct up_session *up_session_find_by_up_seid(struct up_peer *peer, uint64_t up_seid);
 struct up_session *up_session_find_by_cp_f_seid(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid);
 struct up_session *up_session_find_by_local_teid(struct up_peer *peer, uint32_t teid);
@@ -96,8 +94,8 @@ struct pdr {
 
 	bool rx_decaps;
 	bool forw_encaps;
-	bool forw_to_core;
-	bool forw_from_core;
+	bool access_to_core;
+	bool core_to_access;
 
 	struct pdr *reverse_pdr;
 	bool active;

include/osmocom/upf/upf.h

@@ -37,14 +37,21 @@ struct nft_ctx;
 
 #define UPF_PFCP_LISTEN_DEFAULT "0.0.0.0"
 
+#define PORT_GTP0_C 3386
+#define PORT_GTP0_U 3386
+
+#define PORT_GTP1_C 2123
+#define PORT_GTP1_U 2152
+
 extern struct osmo_tdef_group g_upf_tdef_groups[];
+extern struct osmo_tdef g_upf_nft_tdefs[];
 
 struct pfcp_vty_cfg {
 	char *local_addr;
 	uint16_t local_port;
 };
 
-struct gtp_vty_cfg_dev {
+struct tunend_vty_cfg_dev {
 	struct llist_head entry;
 
 	/* If true, osmo-upf creates the GTP device on startup. If false, the GTP device was created by the user, and we
@@ -60,12 +67,18 @@ struct gtp_vty_cfg_dev {
 	char *local_addr;
 };
 
-struct gtp_vty_cfg {
-	/* list of struct gtp_vty_cfg_dev, GTP devices as in the config file. The actual GTP devices in use are in
-	 * g_upf->gtp.devs. */
+struct tunend_vty_cfg {
+	/* list of struct tunend_vty_cfg_dev, GTP devices as in the config file. The actual GTP devices in use are in
+	 * g_upf->tunend.devs. */
 	struct llist_head devs;
 };
 
+/* Item in an llist of string pointers */
+struct string_listitem {
+	struct llist_head entry;
+	char *str;
+};
+
 struct g_upf {
 	struct ctrl_handle *ctrl;
 
@@ -80,7 +93,7 @@ struct g_upf {
 		bool mockup;
 
 		/* GTP devices as in osmo-upf.cfg */
-		struct gtp_vty_cfg vty_cfg;
+		struct tunend_vty_cfg vty_cfg;
 
 		/* GTP devices actually in use, list of struct upf_gtp_dev. */
 		struct llist_head devs;
@@ -89,7 +102,7 @@ struct g_upf {
 		int32_t genl_id;
 
 		uint8_t recovery_count;
-	} gtp;
+	} tunend;
 
 	/* Tunnel forwarding via linux netfilter */
 	struct {
@@ -98,9 +111,16 @@ struct g_upf {
 
 		struct nft_ctx *nft_ctx;
 		char *table_name;
-		int priority;
-		uint32_t next_id_state;
-	} nft;
+		int priority_pre;
+		int priority_post;
+		uint32_t next_chain_id_state;
+	} tunmap;
+
+	struct {
+		uint32_t next_local_teid_state;
+	} gtp;
+
+	struct llist_head netinst;
 };
 
 extern struct g_upf *g_upf;
@@ -115,7 +135,11 @@ enum upf_log_subsys {
 
 void g_upf_alloc(void *ctx);
 void upf_vty_init();
-int upf_pfcp_listen();
+int upf_pfcp_init(void);
+int upf_pfcp_listen(void);
 
 int upf_gtp_devs_open();
 void upf_gtp_devs_close();
+
+uint32_t upf_next_local_teid(void);
+uint32_t upf_next_chain_id(void);

include/osmocom/upf/upf_gtp.h

@@ -27,15 +27,11 @@
 #include <osmocom/core/select.h>
 #include <osmocom/core/logging.h>
 
+#include <osmocom/upf/upf_tun.h>
+
 #define LOG_GTP_DEV(DEV, LEVEL, FMT, ARGS...) \
 	LOGP(DGTP, LEVEL, "%s: " FMT, upf_gtp_dev_to_str_c(OTC_SELECT, (DEV)), ##ARGS)
 
-#define PORT_GTP0_C 3386
-#define PORT_GTP0_U 3386
-
-#define PORT_GTP1_C 2123
-#define PORT_GTP1_U 2152
-
 struct upf_gtp_dev {
 	struct llist_head entry;
 
@@ -57,29 +53,32 @@ struct upf_gtp_dev {
 
 	uint32_t ifidx;
 
+	/* list of struct upf_gtp_tunend */
 	struct llist_head tunnels;
 };
 
-struct upf_gtp_tun_desc {
-	uint32_t local_teid;
-	uint32_t remote_teid;
-	struct osmo_sockaddr ue_addr;
-	struct osmo_sockaddr gtp_remote_addr;
+/* Description of a GTP encapsulation / decapsulation.
+ * The active state to operate the GTP kernel module accordingly is kept in struct upf_gtp_tunend. */
+struct upf_tunend {
+	struct upf_tun access;
+	struct {
+		struct osmo_sockaddr ue_local_addr;
+	} core;
 };
 
-int upf_gtp_tun_desc_cmp(const struct upf_gtp_tun_desc *a, const struct upf_gtp_tun_desc *b);
+int upf_gtp_tunend_cmp(const struct upf_tunend *a, const struct upf_tunend *b);
 
-int upf_gtp_genl_open();
+int upf_gtp_genl_ensure_open();
 void upf_gtp_genl_close();
 
 int upf_gtp_dev_open(const char *name, bool create_gtp_dev, const char *local_addr, bool listen_for_gtpv0,
 		     bool sgsn_mode);
 struct upf_gtp_dev *upf_gtp_dev_find_by_name(const char *name);
+struct upf_gtp_dev *upf_gtp_dev_find_by_local_addr(const struct osmo_sockaddr *local_addr);
 struct upf_gtp_dev *upf_gtp_dev_first();
 
-int upf_gtp_dev_tunnel_add(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *t);
-bool upf_gtp_dev_is_tunnel_active(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *t);
-int upf_gtp_dev_tunnel_del(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *t);
+int upf_gtp_dev_tunend_add(struct upf_gtp_dev *dev, const struct upf_tunend *t);
+int upf_gtp_dev_tunend_del(struct upf_gtp_dev *dev, const struct upf_tunend *t);
 
 int upf_gtp_dev_to_str_buf(char *buf, size_t buflen, const struct upf_gtp_dev *dev);
 char *upf_gtp_dev_to_str_c(void *ctx, const struct upf_gtp_dev *dev);

include/osmocom/upf/upf_nft.h

@@ -25,26 +25,27 @@
 
 #include <stdint.h>
 
-#include <osmocom/core/socket.h>
+#include <osmocom/upf/upf_tun.h>
 
-#define NFT_CHAIN_NAME_PREFIX_TUNMAP "tunmap"
-
-struct upf_nft_tunmap_desc {
-	struct {
-		struct osmo_sockaddr gtp_remote_addr;
-		uint32_t local_teid;
-		uint32_t remote_teid;
-	} access;
-	struct {
-		struct osmo_sockaddr gtp_remote_addr;
-		uint32_t local_teid;
-		uint32_t remote_teid;
-	} core;
-	uint32_t id;
+struct upf_nft_tun {
+	struct upf_tun tun;
+	uint32_t chain_id;
 };
 
+struct upf_tunmap {
+	struct upf_nft_tun access;
+	struct upf_nft_tun core;
+};
+
+int upf_nft_tunmap_to_str_buf(char *buf, size_t buflen, const struct upf_tunmap *tunmap);
+char *upf_nft_tunmap_to_str_c(void *ctx, const struct upf_tunmap *tunmap);
+
 int upf_nft_init();
 int upf_nft_free();
 
-int upf_nft_tunmap_create(struct upf_nft_tunmap_desc *tunmap);
-int upf_nft_tunmap_delete(struct upf_nft_tunmap_desc *tunmap);
+char *upf_nft_tunmap_get_table_init_str(void *ctx);
+char *upf_nft_tunmap_get_vmap_init_str(void *ctx);
+char *upf_nft_tunmap_get_ruleset_str(void *ctx, struct upf_tunmap *tunmap);
+char *upf_nft_tunmap_get_ruleset_del_str(void *ctx, struct upf_tunmap *tunmap);
+int upf_nft_tunmap_create(struct upf_tunmap *tunmap);
+int upf_nft_tunmap_delete(struct upf_tunmap *tunmap);

include/osmocom/upf/upf_tun.h

@@ -0,0 +1,38 @@
+/*
+ * (C) 2023 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ * All Rights Reserved.
+ *
+ * Author: Neels Janosch Hofmeyr <nhofmeyr@sysmocom.de>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#pragma once
+
+#include <stdint.h>
+
+#include <osmocom/core/socket.h>
+
+struct upf_tun_ep {
+	struct osmo_sockaddr addr;
+	uint32_t teid;
+};
+
+struct upf_tun {
+	struct upf_tun_ep local;
+	struct upf_tun_ep remote;
+};

osmoappdesc.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# (C) 2021-2022 by sysmocom - s.m.f.c. GmbH <info@sysmocom.de>
+# (C) 2021-2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or

src/osmo-pfcp-tool/Makefile.am

@@ -10,7 +10,6 @@ AM_CFLAGS = \
 	$(LIBOSMOCORE_CFLAGS) \
 	$(LIBOSMOVTY_CFLAGS) \
 	$(LIBOSMOCTRL_CFLAGS) \
-	$(LIBOSMOGTLV_CFLAGS) \
 	$(LIBOSMOPFCP_CFLAGS) \
 	$(COVERAGE_CFLAGS) \
 	$(NULL)
@@ -19,15 +18,10 @@ AM_LDFLAGS = \
 	$(LIBOSMOCORE_LIBS) \
 	$(LIBOSMOVTY_LIBS) \
 	$(LIBOSMOCTRL_LIBS) \
-	$(LIBOSMOGTLV_LIBS) \
 	$(LIBOSMOPFCP_LIBS) \
 	$(COVERAGE_LDFLAGS) \
 	$(NULL)
 
-noinst_HEADERS = \
-	pfcp_tool.h \
-	$(NULL)
-
 bin_PROGRAMS = \
 	osmo-pfcp-tool \
 	$(NULL)
@@ -36,4 +30,5 @@ osmo_pfcp_tool_SOURCES = \
 	osmo_pfcp_tool_main.c \
 	pfcp_tool.c \
 	pfcp_tool_vty.c \
+	gtp_probe.c \
 	$(NULL)

src/osmo-pfcp-tool/gtp_probe.c

@@ -0,0 +1,233 @@
+/*
+ * (C) 2024 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ * All Rights Reserved.
+ *
+ * Author: Neels Janosch Hofmeyr <nhofmeyr@sysmocom.de>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <errno.h>
+#include <unistd.h>
+#include <netinet/ip.h>
+#include <netinet/udp.h>
+
+#include <osmocom/core/logging.h>
+
+#include <osmocom/pfcptool/gtp_probe.h>
+#include <osmocom/pfcptool/pfcp_tool.h>
+
+void gtp_probe_target_close(struct gtp_probe_target *t)
+{
+	if (t->ofd.fd > 0) {
+		osmo_fd_unregister(&t->ofd);
+		close(t->ofd.fd);
+	}
+	t->ofd.fd = -1;
+	t->addr = (struct osmo_sockaddr){};
+}
+
+static int gtp_probe_cb(struct osmo_fd *ofd, unsigned int what)
+{
+	int rc;
+	struct gtp_probe_state *s = ofd->data;
+	struct osmo_sockaddr remote;
+	socklen_t remote_len = sizeof(remote);
+	struct osmo_sockaddr_str remote_str;
+	char strbuf[64];
+
+	uint8_t rxbuf[2048];
+	struct udphdr *h = (void *)rxbuf;
+
+	if (!(what & OSMO_FD_READ))
+		return 0;
+
+	rc = recvfrom(ofd->fd, rxbuf, sizeof(rxbuf), 0, (struct sockaddr *)&remote, &remote_len);
+	if (rc <= 0)
+		return -EIO;
+
+	osmo_sockaddr_str_from_sockaddr(&remote_str, &remote.u.sas);
+
+	size_t len = ntohl(h->len);
+	uint8_t *data = rxbuf + sizeof(*h);
+	size_t data_len = len - sizeof(*h);
+
+	if (ofd->priv_nr)
+		osmo_escape_cstr_buf(strbuf, sizeof(strbuf), (char *)data, data_len);
+	else
+		osmo_hexdump_buf(strbuf, sizeof(strbuf), data, data_len, NULL, false);
+
+	LOGP(DLGLOBAL, LOGL_NOTICE,
+	     "Probe RX: " OSMO_SOCKADDR_STR_FMT " -> " OSMO_SOCKADDR_STR_FMT " '%s'\n",
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&remote_str),
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&s->ip_target.cfg),
+	     strbuf);
+
+	return 0;
+}
+
+int gtp_probe_target_bind(struct gtp_probe_state *s, struct gtp_probe_target *t)
+{
+	int rc;
+	/* close the existing socket, if any */
+	gtp_probe_target_close(t);
+
+	/* create the new socket, binding to configured local address */
+	t->ofd.data = s;
+	t->ofd.cb = gtp_probe_cb;
+	osmo_sockaddr_str_to_sockaddr(&t->cfg, &t->addr.u.sas);
+	rc = osmo_sock_init_osa_ofd(&t->ofd, SOCK_DGRAM, IPPROTO_UDP, &t->addr, NULL, OSMO_SOCK_F_BIND);
+	if (rc < 0) {
+		t->addr = (struct osmo_sockaddr){};
+		return rc;
+	}
+	LOGP(DLGLOBAL, LOGL_NOTICE, "Bound probe port: " OSMO_SOCKADDR_STR_FMT "\n",
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&t->cfg));
+	return 0;
+}
+
+/* According to 3GPP TS 29.060. */
+struct gtp1u_hdr {
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+	uint8_t pn:1, s:1, e:1, spare:1, pt:1, version:3;
+#else
+	uint8_t version:3, pt:1, spare:1, e:1, s:1, pn:1;
+#endif
+	uint8_t type;
+	uint16_t length;
+	uint32_t tei;
+};
+
+struct gtp_pdu {
+	struct gtp1u_hdr gtp;
+	struct iphdr ip;
+	struct udphdr udp;
+	char data[1024];
+} __attribute__ ((packed));
+
+int gtp_probe_run_tunend(struct gtp_probe_state *s, struct pfcp_tool_gtp_tun *inject_gtp,
+			 struct osmo_sockaddr_str *ue_ip,
+			 struct gtp_probe_target *expect_rx_on_ip)
+{
+	int rc;
+	/* remote-GTP           this session's ACCESS side            gtp_probe UDP
+	 *    |--------UDP/GTP---->|--------------------------->[...]--->|
+	 *    |                    |
+	 * inject                  |                                  expect UDP
+	 *  from                   |                                  at the ip_target
+	 * remote-GTP-IP:2152   to UPF's F-TEID
+	 */
+	//struct osmo_sockaddr_str *remote_gtp = &inject_gtp->remote.addr;
+	struct osmo_sockaddr_str upf_gtp = inject_gtp->local.addr;
+	upf_gtp.port = 2152;
+	struct osmo_sockaddr sendto_addr;
+	uint32_t upf_rx_teid = inject_gtp->local.teid;
+	struct osmo_sockaddr_str *payload_dst = &expect_rx_on_ip->cfg;
+	struct osmo_sockaddr_str payload_src = *ue_ip;
+	payload_src.port = payload_dst->port;
+
+	struct gtp_pdu pdu = {
+		.gtp = {
+			.pn = 0,
+			.s = 0,
+			.e = 0,
+			.spare = 0,
+			.pt = 1,
+			.version = 1,
+			.type = 0xff,	/* G-PDU */
+			.length = 0,	/* filled in later */
+			.tei = htonl(upf_rx_teid),
+		},
+		.ip = {
+			.ihl = 5,
+			.version = 4,
+			.id = 1234, // TODO
+			.frag_off = htons(0x4000),
+			.ttl = 64,
+			.protocol = IPPROTO_UDP,
+			.check = 0, // TODO
+		},
+		.udp = {
+			.source = htons(payload_src.port),
+			.dest = htons(payload_dst->port),
+			.check = 0, // TODO
+		},
+		.data = "hello!",
+	};
+
+	size_t data_len = strlen(pdu.data) + 1;
+	size_t udp_len = sizeof(pdu.udp) + data_len;
+	size_t ip_len = sizeof(pdu.ip) + udp_len;
+	size_t gtp_len = sizeof(pdu.gtp) + ip_len;
+	uint32_t i32;
+	pdu.gtp.length = htons(gtp_len);
+	pdu.ip.tot_len = htons(ip_len);
+	pdu.udp.len = htons(udp_len);
+
+	LOGP(DLGLOBAL, LOGL_NOTICE, "GTP-probe: sending payload GTP{%zu IP{%zu UDP{%zu {%zu %s} } } }\n",
+	     gtp_len, ip_len, udp_len, data_len, osmo_quote_cstr_c(OTC_SELECT, pdu.data, data_len));
+
+	rc = osmo_sockaddr_str_to_32(&payload_src, &i32);
+	if (rc) {
+		LOGP(DLGLOBAL, LOGL_ERROR, "invalid payload source address: " OSMO_SOCKADDR_STR_FMT "\n",
+		     OSMO_SOCKADDR_STR_FMT_ARGS(&payload_src));
+		return rc;
+	}
+	pdu.ip.saddr = i32;
+
+	rc = osmo_sockaddr_str_to_32(payload_dst, &i32);
+	if (rc) {
+		LOGP(DLGLOBAL, LOGL_ERROR, "invalid payload destination address: " OSMO_SOCKADDR_STR_FMT "\n",
+		     OSMO_SOCKADDR_STR_FMT_ARGS(payload_dst));
+		return rc;
+	}
+	pdu.ip.daddr = i32;
+
+#if 0
+	/* Skip GTP tunnel for debugging */
+	upf_gtp = *payload_dst;
+#endif
+
+	rc = osmo_sockaddr_str_to_sockaddr(&upf_gtp, &sendto_addr.u.sas);
+	if (rc) {
+		LOGP(DLGLOBAL, LOGL_ERROR, "invalid UPF GTP address: " OSMO_SOCKADDR_STR_FMT "\n",
+		     OSMO_SOCKADDR_STR_FMT_ARGS(&upf_gtp));
+		return rc;
+	}
+
+	LOGP(DLGLOBAL, LOGL_INFO, "GTP-probe sending: " OSMO_SOCKADDR_STR_FMT " -> " OSMO_SOCKADDR_STR_FMT ":TEID-0x%x:"
+	     " payload: UDP " OSMO_SOCKADDR_STR_FMT " -> " OSMO_SOCKADDR_STR_FMT "\n",
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&s->gtp_target.cfg),
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&upf_gtp),
+	     upf_rx_teid,
+	     OSMO_SOCKADDR_STR_FMT_ARGS(&payload_src),
+	     OSMO_SOCKADDR_STR_FMT_ARGS(payload_dst));
+
+	// FIXME: allow more than just one gtp_target, and figure out here which one to use from remote_gtp above.
+	rc = sendto(s->gtp_target.ofd.fd, (void *)&pdu, gtp_len, 0,
+		    &sendto_addr.u.sa, sizeof(sendto_addr));
+	if (rc != gtp_len) {
+		LOGP(DLGLOBAL, LOGL_ERROR, "%s.sendto(%s) failed: rc = %d != length %zu; errno = %d %s\n",
+		     osmo_sock_get_name(OTC_SELECT, s->gtp_target.ofd.fd),
+		     osmo_sockaddr_to_str_c(OTC_SELECT, &sendto_addr), rc, gtp_len,
+		     errno, strerror(errno));
+		return -EIO;
+	}
+
+
+	return 0;
+}

src/osmo-pfcp-tool/osmo_pfcp_tool_main.c

@@ -42,7 +42,7 @@
 
 #include <osmocom/pfcp/pfcp_endpoint.h>
 
-#include "pfcp_tool.h"
+#include <osmocom/pfcptool/pfcp_tool.h>
 
 #define _GNU_SOURCE
 #include <getopt.h>
@@ -206,17 +206,16 @@ static void signal_handler(int signum)
 	}
 }
 
-static const char * const osmo_pfcp_tool_copyright =
-	"OsmoPFCPTool - Osmocom Packet Forwarding Control Protocol tool for testing\r\n"
-	"Copyright (C) 2021-2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>\r\n"
-	"License AGPLv3+: GNU AGPL version 3 or later <http://gnu.org/licenses/agpl-3.0.html>\r\n"
-	"This is free software: you are free to change and redistribute it.\r\n"
-	"There is NO WARRANTY, to the extent permitted by law.\r\n";
 
 static struct vty_app_info pfcp_tool_vty_app_info = {
 	.name = "osmo-pfcp-tool",
 	.version = PACKAGE_VERSION,
-	.copyright = osmo_pfcp_tool_copyright,
+	.copyright =
+	"OsmoPFCPTool - Osmocom Packet Forwarding Control Protocol tool for testing\r\n"
+	"Copyright (C) 2021-2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>\r\n"
+	"License AGPLv3+: GNU AGPL version 3 or later <http://gnu.org/licenses/agpl-3.0.html>\r\n"
+	"This is free software: you are free to change and redistribute it.\r\n"
+	"There is NO WARRANTY, to the extent permitted by law.\r\n",
 };
 
 static const struct log_info_cat pfcp_tool_default_categories[] = {
@@ -227,14 +226,15 @@ const struct log_info log_info = {
 	.num_cat = ARRAY_SIZE(pfcp_tool_default_categories),
 };
 
-int pfcp_tool_mainloop()
+int pfcp_tool_mainloop(int poll)
 {
+	int rc;
 	log_reset_context();
-	osmo_select_main_ctx(0);
+	rc = osmo_select_main_ctx(poll);
 
 	/* If the user hits Ctrl-C the third time, just terminate immediately. */
 	if (quit >= 3)
-		return 1;
+		return -1;
 
 	/* Has SIGTERM been received (and not yet been handled)? */
 	if (quit && !osmo_select_shutdown_requested()) {
@@ -244,7 +244,7 @@ int pfcp_tool_mainloop()
 		osmo_select_shutdown_request();
 		/* continue the main select loop until all write queues are serviced. */
 	}
-	return 0;
+	return rc;
 }
 
 int main(int argc, char **argv)
@@ -300,13 +300,13 @@ int main(int argc, char **argv)
 		}
 	}
 
-	/* start telnet, after reading config for vty_get_bind_addr() */
-	rc = telnet_init_dynif(tall_pfcp_tool_ctx, &g_pfcp_tool, vty_get_bind_addr(), OSMO_VTY_PORT_PFCP_TOOL);
+	/* start telnet VTY */
+	rc = telnet_init_default(tall_pfcp_tool_ctx, &g_pfcp_tool, OSMO_VTY_PORT_PFCP_TOOL);
 	if (rc < 0)
 		return 2;
 
 	/* start control interface, after reading config for ctrl_vty_get_bind_addr() */
-	g_pfcp_tool->ctrl = ctrl_interface_setup_dynip(g_pfcp_tool, ctrl_vty_get_bind_addr(), OSMO_CTRL_PORT_PFCP_TOOL, NULL);
+	g_pfcp_tool->ctrl = ctrl_interface_setup(g_pfcp_tool, OSMO_CTRL_PORT_PFCP_TOOL, NULL);
 	if (!g_pfcp_tool->ctrl) {
 		fprintf(stderr, "Failed to initialize control interface. Exiting.\n");
 		return -1;
@@ -327,7 +327,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	pfcp_tool_mainloop();
+	pfcp_tool_mainloop(1);
 
 	pfcp_tool_vty_init_cmds();
 
@@ -342,14 +342,14 @@ int main(int argc, char **argv)
 		printf("Done reading '%s', waiting for retransmission queue...\n",
 		       pfcp_tool_cmdline_config.command_file);
 		do {
-			if (pfcp_tool_mainloop())
+			if (pfcp_tool_mainloop(0) == -1)
 				break;
 		} while (osmo_pfcp_endpoint_retrans_queue_is_busy(g_pfcp_tool->ep));
 		printf("Done\n");
 	} else {
 		printf("Listening for commands on VTY...\n");
 		do {
-			if (pfcp_tool_mainloop())
+			if (pfcp_tool_mainloop(0) == -1)
 				break;
 		} while (!osmo_select_shutdown_done());
 	}

src/osmo-pfcp-tool/pfcp_tool.c

@@ -26,7 +26,7 @@
 
 #include <osmocom/pfcp/pfcp_endpoint.h>
 
-#include "pfcp_tool.h"
+#include <osmocom/pfcptool/pfcp_tool.h>
 
 struct g_pfcp_tool *g_pfcp_tool = NULL;
 
@@ -48,6 +48,8 @@ void g_pfcp_tool_alloc(void *ctx)
 	};
 
 	INIT_LLIST_HEAD(&g_pfcp_tool->peers);
+
+	g_pfcp_tool->next_ue_addr_state = 0xc0a90001;
 }
 
 struct pfcp_tool_peer *pfcp_tool_peer_find(const struct osmo_sockaddr *remote_addr)
@@ -85,7 +87,7 @@ struct pfcp_tool_session *pfcp_tool_session_find(struct pfcp_tool_peer *peer, ui
 }
 
 struct pfcp_tool_session *pfcp_tool_session_find_or_create(struct pfcp_tool_peer *peer, uint64_t cp_seid,
-							   enum up_gtp_action_kind gtp_action)
+							   enum up_gtp_action_kind kind)
 {
 	struct pfcp_tool_session *session = pfcp_tool_session_find(peer, cp_seid);
 	if (session)
@@ -95,7 +97,7 @@ struct pfcp_tool_session *pfcp_tool_session_find_or_create(struct pfcp_tool_peer
 	*session = (struct pfcp_tool_session){
 		.peer = peer,
 		.cp_seid = cp_seid,
-		.gtp_action = gtp_action,
+		.kind = kind,
 	};
 	llist_add(&session->entry, &peer->sessions);
 	return session;
@@ -144,6 +146,32 @@ static void rx_session_est_resp(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_
 	if (!session)
 		return;
 	session->up_f_seid = m->ies.session_est_resp.up_f_seid;
+
+	/* Retrieve the local-F-TEIDs that the UPF has created */
+	for (int i = 0; i < m->ies.session_est_resp.created_pdr_count; i++) {
+		struct osmo_pfcp_ie_created_pdr *cpdr = &m->ies.session_est_resp.created_pdr[i];
+		struct pfcp_tool_gtp_tun_ep *target = NULL;
+
+		if (!cpdr->local_f_teid_present)
+			continue;
+		if (cpdr->local_f_teid.choose_flag)
+			continue;
+
+		switch (session->kind) {
+		case UP_GTP_U_TUNEND:
+			if (session->tunend.access.local_pdr_id == cpdr->pdr_id)
+				target = &session->tunend.access.local;
+			break;
+		default:
+			OSMO_ASSERT(0);
+			// TODO
+		}
+
+		if (target) {
+			osmo_pfcp_ip_addrs_get_str(&target->addr, NULL, &cpdr->local_f_teid.fixed.ip_addr);
+			target->teid = cpdr->local_f_teid.fixed.teid;
+		}
+	}
 }
 
 void pfcp_tool_rx_msg(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_msg *m, struct osmo_pfcp_msg *req)
@@ -159,14 +187,27 @@ void pfcp_tool_rx_msg(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_msg *m, st
 	}
 }
 
+static void copy_msg(struct osmo_pfcp_msg *dst, const struct osmo_pfcp_msg *m)
+{
+	*dst = *m;
+	dst->encoded = NULL;
+	dst->ctx.peer_use_token = NULL;
+	dst->ctx.session_use_token = NULL;
+	dst->ctx.resp_cb = NULL;
+}
+
 int peer_tx(struct pfcp_tool_peer *peer, struct osmo_pfcp_msg *m)
 {
 	int rc;
-	rc = osmo_pfcp_endpoint_tx(g_pfcp_tool->ep, m);
 	if (m->is_response)
-		peer->last_resp = *m;
+		copy_msg(&peer->last_resp, m);
 	else
-		peer->last_req = *m;
+		copy_msg(&peer->last_req, m);
+	rc = osmo_pfcp_endpoint_tx(g_pfcp_tool->ep, m);
+	if (rc) {
+		LOGP(DLPFCP, LOGL_ERROR, "Failed to transmit PFCP: %s\n", strerror(-rc));
+		return CMD_WARNING;
+	}
 	return rc;
 }
 
@@ -174,3 +215,14 @@ uint64_t peer_new_seid(struct pfcp_tool_peer *peer)
 {
 	return peer->next_seid_state++;
 }
+
+uint32_t pfcp_tool_new_teid(void)
+{
+	return g_pfcp_tool->next_teid_state++;
+}
+
+int pfcp_tool_next_ue_addr(struct osmo_sockaddr_str *dst)
+{
+	uint32_t addr = g_pfcp_tool->next_ue_addr_state++;
+	return osmo_sockaddr_str_from_32h(dst, addr, 0);
+}

src/osmo-upf/Makefile.am

@@ -10,7 +10,6 @@ AM_CFLAGS = \
 	$(LIBOSMOCORE_CFLAGS) \
 	$(LIBOSMOVTY_CFLAGS) \
 	$(LIBOSMOCTRL_CFLAGS) \
-	$(LIBOSMOGTLV_CFLAGS) \
 	$(LIBOSMOPFCP_CFLAGS) \
 	$(LIBGTPNL_CFLAGS) \
 	$(LIBNFTNL_CFLAGS) \
@@ -19,18 +18,15 @@ AM_CFLAGS = \
 	$(NULL)
 
 AM_LDFLAGS = \
-	$(LIBGTPNL_LDFLAGS) \
-	$(LIBNFTNL_LDFLAGS) \
-	$(LIBNFTABLES_LDFLAGS) \
 	$(COVERAGE_LDFLAGS) \
 	$(NULL)
 
-bin_PROGRAMS = \
-	osmo-upf \
+noinst_LTLIBRARIES = \
+	libupf.la \
 	$(NULL)
 
-osmo_upf_SOURCES = \
-	osmo_upf_main.c \
+libupf_la_SOURCES = \
+	netinst.c \
 	up_endpoint.c \
 	up_gtp_action.c \
 	up_peer.c \
@@ -42,14 +38,24 @@ osmo_upf_SOURCES = \
 	upf_vty.c \
 	$(NULL)
 
-osmo_upf_LDADD = \
+libupf_la_LIBADD = \
 	$(LIBOSMOCORE_LIBS) \
 	$(LIBOSMOVTY_LIBS) \
 	$(LIBOSMOCTRL_LIBS) \
-	$(LIBOSMOGTLV_LIBS) \
 	$(LIBOSMOPFCP_LIBS) \
 	$(LIBGTPNL_LIBS) \
 	$(LIBNFTNL_LIBS) \
 	$(LIBNFTABLES_LIBS) \
-	$(COVERAGE_LDFLAGS) \
+	$(NULL)
+
+bin_PROGRAMS = \
+	osmo-upf \
+	$(NULL)
+
+osmo_upf_SOURCES = \
+	osmo_upf_main.c \
+	$(NULL)
+
+osmo_upf_LDADD = \
+	libupf.la \
 	$(NULL)

src/osmo-upf/netinst.c

@@ -0,0 +1,124 @@
+/*
+ * (C) 2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ * All Rights Reserved.
+ *
+ * Author: Neels Janosch Hofmeyr <nhofmeyr@sysmocom.de>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <string.h>
+
+#include <osmocom/core/talloc.h>
+#include <osmocom/core/logging.h>
+#include <osmocom/vty/vty.h>
+
+#include <osmocom/upf/netinst.h>
+
+/* Add a new netinst entry to the given list.
+ * \param ctx  talloc allocate new entry from ctx.
+ * \param list  append to this list.
+ * \param name  The Network Instance name as given in PFCP Network Instance IEs.
+ * \param addr  IP address string of local interface to associate with the Network Instance.
+ * \param errmsg  On error, an error description is returned in this out-argument.
+ * \return new network_instance entry, or NULL on error.
+ */
+const struct network_instance *netinst_add(void *ctx, struct llist_head *list, const char *name, const char *addr,
+					   const char **errmsg)
+{
+	struct network_instance *netinst;
+	if (errmsg)
+		*errmsg = NULL;
+
+	if (!name || !*name) {
+		if (errmsg)
+			*errmsg = "Network Instance name must not be empty";
+		return NULL;
+	}
+
+	if (netinst_find(list, name)) {
+		if (errmsg)
+			*errmsg = "Network Instance entry with this name already exists";
+		return NULL;
+	}
+
+	netinst = talloc(ctx, struct network_instance);
+	*netinst = (struct network_instance){
+		.name = talloc_strdup(netinst, name),
+	};
+	if (osmo_sockaddr_str_from_str(&netinst->addr, addr, 0)) {
+		if (errmsg)
+			*errmsg = "Network Instance address is not a valid IP address string";
+		talloc_free(netinst);
+		return NULL;
+	}
+
+	llist_add_tail(&netinst->entry, list);
+
+	return netinst;
+}
+
+const struct network_instance *netinst_find(struct llist_head *list, const char *name)
+{
+	const struct network_instance *netinst;
+
+	if (!name)
+		return NULL;
+
+	llist_for_each_entry(netinst, list, entry)
+		if (!strcmp(netinst->name, name))
+			return netinst;
+
+	return NULL;
+}
+
+const struct network_instance *netinst_first(struct llist_head *list)
+{
+	return llist_first_entry_or_null(list, struct network_instance, entry);
+}
+
+/* Clear the list of Network Instance entries, return the nr of entries that were removed. */
+int netinst_clear(struct llist_head *list)
+{
+	int count = 0;
+	while (1) {
+		struct network_instance *netinst = llist_first_entry_or_null(list, struct network_instance, entry);
+		if (!netinst)
+			break;
+		llist_del(&netinst->entry);
+		talloc_free(netinst);
+		count++;
+	}
+	return count;
+}
+
+/* Write one or all netinst entries to the VTY output.
+ * If name_or_null is NULL, print all entries. Else, print only the entry matching that name.
+ * Return number of printed entries. */
+int netinst_vty_write(struct vty *vty, struct llist_head *list, const char *indent, const char *name_or_null)
+{
+	const struct network_instance *netinst;
+	int count = 0;
+
+	llist_for_each_entry(netinst, list, entry) {
+		if (name_or_null && strcmp(netinst->name, name_or_null))
+			continue;
+		vty_out(vty, "%sadd %s %s%s", indent, netinst->name, netinst->addr.ip, VTY_NEWLINE);
+		count++;
+	}
+	return count;
+}

src/osmo-upf/osmo_upf_main.c

@@ -304,13 +304,13 @@ int main(int argc, char **argv)
 		return 1;
 	}
 
-	/* start telnet, after reading config for vty_get_bind_addr() */
-	rc = telnet_init_dynif(tall_upf_ctx, &g_upf, vty_get_bind_addr(), OSMO_VTY_PORT_UPF);
+	/* start telnet VTY */
+	rc = telnet_init_default(tall_upf_ctx, &g_upf, OSMO_VTY_PORT_UPF);
 	if (rc < 0)
 		return 2;
 
 	/* start control interface, after reading config for ctrl_vty_get_bind_addr() */
-	g_upf->ctrl = ctrl_interface_setup_dynip(g_upf, ctrl_vty_get_bind_addr(), OSMO_CTRL_PORT_UPF, NULL);
+	g_upf->ctrl = ctrl_interface_setup(g_upf, OSMO_CTRL_PORT_UPF, NULL);
 	if (!g_upf->ctrl) {
 		fprintf(stderr, "Failed to initialize control interface. Exiting.\n");
 		return -1;
@@ -331,9 +331,6 @@ int main(int argc, char **argv)
 		}
 	}
 
-	if (upf_gtp_genl_open())
-		return -1;
-
 	if (upf_gtp_devs_open())
 		return -1;
 

src/osmo-upf/up_endpoint.c

@@ -38,7 +38,7 @@ static void up_endpoint_set_msg_ctx(struct osmo_pfcp_endpoint *ep, struct osmo_p
 		if (!m->ctx.peer_fi && req->ctx.peer_fi)
 			up_peer_set_msg_ctx(req->ctx.peer_fi->priv, m);
 		if (!m->ctx.session_fi && req->ctx.session_fi)
-			up_session_set_msg_ctx(req->ctx.peer_fi->priv, m);
+			up_session_set_msg_ctx(req->ctx.session_fi->priv, m);
 	}
 
 	/* From the remote address, find the matching peer instance */
@@ -221,15 +221,20 @@ static void up_endpoint_rx_cb(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_ms
 	case OSMO_PFCP_MSGT_SESSION_REP_REQ:
 		up_ep_rx_session_rep_req(up_ep, m);
 		return;
+	case OSMO_PFCP_MSGT_HEARTBEAT_REQ:
+	case OSMO_PFCP_MSGT_HEARTBEAT_RESP:
+		/* Heartbeat is already handled in osmo_pfcp_endpoint_handle_rx() in pfcp_endpoint.c. The heartbeat
+		 * messages are also dispatched here, to the rx_cb, "on informtional basis", nothing needs to happen
+		 * here. */
+		return;
 	default:
 		OSMO_LOG_PFCP_MSG(m, LOGL_ERROR, "Unknown message type\n");
 		return;
 	}
 }
 
-struct up_endpoint *up_endpoint_init(void *ctx, const struct osmo_sockaddr *local_addr)
+struct up_endpoint *up_endpoint_alloc(void *ctx, const struct osmo_sockaddr *local_addr)
 {
-	int rc;
 	struct osmo_pfcp_endpoint_cfg cfg;
 	struct up_endpoint *up_ep;
 	up_ep = talloc_zero(ctx, struct up_endpoint);
@@ -246,14 +251,16 @@ struct up_endpoint *up_endpoint_init(void *ctx, const struct osmo_sockaddr *loca
 	up_ep->pfcp_ep = osmo_pfcp_endpoint_create(up_ep, &cfg);
 	OSMO_ASSERT(up_ep->pfcp_ep);
 
-	rc = osmo_pfcp_endpoint_bind(up_ep->pfcp_ep);
-	if (rc) {
-		talloc_free(up_ep);
-		return NULL;
-	}
 	return up_ep;
 }
 
+int up_endpoint_bind(struct up_endpoint *up_ep)
+{
+	OSMO_ASSERT(up_ep);
+	OSMO_ASSERT(up_ep->pfcp_ep);
+	return osmo_pfcp_endpoint_bind(up_ep->pfcp_ep);
+}
+
 static struct up_session *up_endpoint_find_session(struct up_endpoint *ep, uint64_t up_seid)
 {
 		struct up_peer *peer;
@@ -265,22 +272,11 @@ static struct up_session *up_endpoint_find_session(struct up_endpoint *ep, uint6
 		return NULL;
 }
 
-static struct up_session *up_endpoint_find_session_by_local_teid(struct up_endpoint *ep, uint32_t teid)
-{
-		struct up_peer *peer;
-		llist_for_each_entry(peer, &ep->peers, entry) {
-			struct up_session *session = up_session_find_by_local_teid(peer, teid);
-			if (session)
-				return session;
-		}
-		return NULL;
-}
-
-uint64_t up_endpoint_next_seid(struct up_endpoint *ep)
+uint64_t up_endpoint_next_up_seid(struct up_endpoint *ep)
 {
 	uint64_t sanity;
 	for (sanity = 2342; sanity; sanity--) {
-		uint64_t next_seid = osmo_pfcp_next_seid(&ep->next_seid_state);
+		uint64_t next_seid = osmo_pfcp_next_seid(&ep->next_up_seid_state);
 		if (up_endpoint_find_session(ep, next_seid))
 			continue;
 		return next_seid;
@@ -288,26 +284,6 @@ uint64_t up_endpoint_next_seid(struct up_endpoint *ep)
 	return 0;
 }
 
-static uint32_t up_endpoint_inc_teid(struct up_endpoint *ep)
-{
-	ep->next_teid_state++;
-	if (!ep->next_teid_state)
-		ep->next_teid_state++;
-	return ep->next_teid_state;
-}
-
-uint32_t up_endpoint_next_teid(struct up_endpoint *ep)
-{
-	uint32_t sanity;
-	for (sanity = 2342; sanity; sanity--) {
-		uint32_t next_teid = up_endpoint_inc_teid(ep);
-		if (up_endpoint_find_session_by_local_teid(ep, next_teid))
-			continue;
-		return next_teid;
-	}
-	return 0;
-}
-
 void up_endpoint_free(struct up_endpoint **_ep)
 {
 	struct up_peer *peer;

src/osmo-upf/up_gtp_action.c

@@ -47,27 +47,27 @@ int up_gtp_action_cmp(const struct up_gtp_action *a, const struct up_gtp_action
 		return cmp;
 
 	switch (a->kind) {
-	case UP_GTP_U_ENDECAPS:
-		if ((cmp = CMP_MEMB(endecaps.local_teid)))
+	case UP_GTP_U_TUNEND:
+		if ((cmp = CMP_MEMB(tunend.access.local.teid)))
 			return cmp;
-		if ((cmp = CMP_MEMB(endecaps.remote_teid)))
+		if ((cmp = CMP_MEMB(tunend.access.remote.teid)))
 			return cmp;
-		cmp = osmo_sockaddr_cmp(&a->endecaps.gtp_remote_addr, &b->endecaps.gtp_remote_addr);
+		cmp = osmo_sockaddr_cmp(&a->tunend.access.remote.addr, &b->tunend.access.remote.addr);
 		if (cmp)
 			return cmp;
-		cmp = osmo_sockaddr_cmp(&a->endecaps.ue_addr, &b->endecaps.ue_addr);
+		cmp = osmo_sockaddr_cmp(&a->tunend.core.ue_local_addr, &b->tunend.core.ue_local_addr);
 		if (cmp)
 			return cmp;
 		break;
 
 	case UP_GTP_U_TUNMAP:
-		if ((cmp = CMP_MEMB(tunmap.access.local_teid)))
+		if ((cmp = CMP_MEMB(tunmap.access.tun.local.teid)))
 			return cmp;
-		if ((cmp = CMP_MEMB(tunmap.access.remote_teid)))
+		if ((cmp = CMP_MEMB(tunmap.access.tun.remote.teid)))
 			return cmp;
-		if ((cmp = CMP_MEMB(tunmap.core.local_teid)))
+		if ((cmp = CMP_MEMB(tunmap.core.tun.local.teid)))
 			return cmp;
-		if ((cmp = CMP_MEMB(tunmap.core.remote_teid)))
+		if ((cmp = CMP_MEMB(tunmap.core.tun.remote.teid)))
 			return cmp;
 		break;
 	default:
@@ -79,70 +79,61 @@ int up_gtp_action_cmp(const struct up_gtp_action *a, const struct up_gtp_action
 static int up_gtp_action_enable_disable(struct up_gtp_action *a, bool enable)
 {
 	struct upf_gtp_dev *gtp_dev;
+	const struct osmo_sockaddr *gtp_addr;
 	int rc;
 
 	switch (a->kind) {
-	case UP_GTP_U_ENDECAPS:
-		if (g_upf->gtp.mockup) {
-			LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "gtp/mockup active, skipping GTP action %s\n",
+	case UP_GTP_U_TUNEND:
+		if (g_upf->tunend.mockup) {
+			LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "tunend/mockup active, skipping GTP action %s\n",
 					  enable ? "enable" : "disable");
 			return 0;
 		}
 
-		/* use the first available GTP device.
-		 * TODO: select by interface name?
-		 */
-		gtp_dev = upf_gtp_dev_first();
+		/* Pick GTP device matching the local F-TEID set up for the GTP tunnel (it is on the Access side) */
+		gtp_addr = &a->tunend.access.local.addr;
+		gtp_dev = upf_gtp_dev_find_by_local_addr(gtp_addr);
 		if (!gtp_dev) {
-			LOG_UP_GTP_ACTION(a, LOGL_ERROR, "No GTP device open, cannot %s\n", enable ? "enable" : "disable");
+			LOG_UP_GTP_ACTION(a, LOGL_ERROR, "No GTP device open for local address %s, cannot %s"
+					  " -- consider configuring 'tunend' / 'dev (create|use) foo %s'\n",
+					  osmo_sockaddr_to_str_c(OTC_SELECT, gtp_addr),
+					  enable ? "enable" : "disable",
+					  osmo_sockaddr_to_str_c(OTC_SELECT, gtp_addr));
 			return -EIO;
 		}
 
 		if (enable)
-			rc = upf_gtp_dev_tunnel_add(gtp_dev, &a->endecaps);
+			rc = upf_gtp_dev_tunend_add(gtp_dev, &a->tunend);
 		else
-			rc = upf_gtp_dev_tunnel_del(gtp_dev, &a->endecaps);
+			rc = upf_gtp_dev_tunend_del(gtp_dev, &a->tunend);
 		if (rc) {
-			LOG_UP_GTP_ACTION(a, LOGL_ERROR, "Failed to %s GTP tunnel: %d %s\n",
-					  enable ? "enable" : "disable", rc, strerror(-rc));
+			LOG_UP_GTP_ACTION(a, LOGL_ERROR, "Failed to %s GTP tunnel (rc=%d)\n",
+					  enable ? "enable" : "disable", rc);
 			return rc;
 		}
-		LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "%s GTP tunnel\n", enable ? "Enabled" : "Disabled");
+		LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "%s tunend on dev %s\n", enable ? "Enabled" : "Disabled",
+				  gtp_dev->name);
 		return 0;
 
 	case UP_GTP_U_TUNMAP:
-		if (g_upf->nft.mockup) {
-			LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "nft/mockup active, skipping nftables ruleset %s\n",
+		if (g_upf->tunmap.mockup) {
+			LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "tunmap/mockup active, skipping nftables ruleset %s\n",
 					  enable ? "enable" : "disable");
 			return 0;
 		}
 
-		if (enable && a->tunmap.id != 0) {
-			LOG_UP_GTP_ACTION(a, LOGL_ERROR,
-					  "Cannot enable: nft GTP tunnel mapping rule has been enabled before"
-					  " as " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u\n", a->tunmap.id);
-			return -EALREADY;
-		}
-		if (!enable && a->tunmap.id == 0) {
-			LOG_UP_GTP_ACTION(a, LOGL_ERROR,
-					  "Cannot disable: nft GTP tunnel mapping rule has not been enabled"
-					  " (no " NFT_CHAIN_NAME_PREFIX_TUNMAP " id)\n");
-			return -ENOENT;
-		}
 		if (enable)
 			rc = upf_nft_tunmap_create(&a->tunmap);
 		else
 			rc = upf_nft_tunmap_delete(&a->tunmap);
 		if (rc) {
-			LOG_UP_GTP_ACTION(a, LOGL_ERROR,
-					  "Failed to %s nft GTP tunnel mapping " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u:"
-					  " %d %s\n", enable ? "enable" : "disable", a->tunmap.id, rc, strerror(-rc));
+			LOG_UP_GTP_ACTION(a, LOGL_ERROR, "Failed to %s nft GTP tunnel mapping (rc=%d)\n",
+					  enable ? "enable" : "disable", rc);
 			return rc;
 		}
-		LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "%s nft GTP tunnel mapping " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u\n",
-				  enable ? "Enabled" : "Disabled", a->tunmap.id);
-		if (!enable)
-			a->tunmap.id = 0;
+		LOG_UP_GTP_ACTION(a, LOGL_NOTICE, "%s tunmap, nft chain IDs: access--%u-> <-%u--core\n",
+				  enable ? "Enabled" : "Disabled",
+				  a->tunmap.access.chain_id, a->tunmap.core.chain_id);
 		return 0;
 
 	default:
@@ -165,21 +156,28 @@ int up_gtp_action_to_str_buf(char *buf, size_t buflen, const struct up_gtp_actio
 {
 	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
 	switch (a->kind) {
-	case UP_GTP_U_ENDECAPS:
-		OSMO_STRBUF_PRINTF(sb, "GTP:endecaps GTP-access:");
-		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->endecaps.gtp_remote_addr);
-		OSMO_STRBUF_PRINTF(sb, " TEID-r:0x%"PRIx32" TEID-l:0x%"PRIx32" IP-core:",
-				   a->endecaps.remote_teid, a->endecaps.local_teid);
-		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->endecaps.ue_addr);
+	case UP_GTP_U_TUNEND:
+		OSMO_STRBUF_PRINTF(sb, "GTP:tunend GTP-access-r:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunend.access.remote.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-access-r:0x%"PRIx32, a->tunend.access.remote.teid);
+		OSMO_STRBUF_PRINTF(sb, " GTP-access-l:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunend.access.local.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-access-l:0x%"PRIx32" IP-core-l:", a->tunend.access.local.teid);
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunend.core.ue_local_addr);
 		break;
 	case UP_GTP_U_TUNMAP:
-		OSMO_STRBUF_PRINTF(sb, "GTP:tunmap GTP-access:");
-		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.access.gtp_remote_addr);
-		OSMO_STRBUF_PRINTF(sb, " TEID-access-r:0x%"PRIx32" TEID-access-l:0x%"PRIx32" GTP-core:",
-				   a->tunmap.access.remote_teid, a->tunmap.access.local_teid);
-		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.core.gtp_remote_addr);
-		OSMO_STRBUF_PRINTF(sb, " TEID-core-r:0x%"PRIx32" TEID-core-l:0x%"PRIx32,
-				   a->tunmap.core.remote_teid, a->tunmap.core.local_teid);
+		OSMO_STRBUF_PRINTF(sb, "GTP:tunmap GTP-access-r:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.access.tun.remote.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-access-r:0x%"PRIx32, a->tunmap.access.tun.remote.teid);
+		OSMO_STRBUF_PRINTF(sb, " GTP-access-l:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.access.tun.local.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-access-l:0x%"PRIx32, a->tunmap.access.tun.local.teid);
+		OSMO_STRBUF_PRINTF(sb, " GTP-core-r:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.core.tun.remote.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-core-r:0x%"PRIx32, a->tunmap.core.tun.remote.teid);
+		OSMO_STRBUF_PRINTF(sb, " GTP-core-l:");
+		OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &a->tunmap.core.tun.local.addr);
+		OSMO_STRBUF_PRINTF(sb, " TEID-core-l:0x%"PRIx32, a->tunmap.core.tun.local.teid);
 		break;
 	case UP_GTP_DROP:
 		OSMO_STRBUF_PRINTF(sb, "GTP:drop");
@@ -189,9 +187,10 @@ int up_gtp_action_to_str_buf(char *buf, size_t buflen, const struct up_gtp_actio
 		break;
 	}
 	if (a->session)
-		OSMO_STRBUF_PRINTF(sb, " PFCP-peer:%s SEID-l:0x%"PRIx64" PDR:%d,%d",
-				   up_peer_remote_addr_str(a->session->up_peer),
-				   a->session->up_seid, a->pdr_core, a->pdr_access);
+		OSMO_STRBUF_PRINTF(sb, " PFCP-peer:%s SEID-l:0x%"PRIx64,
+				   up_peer_remote_addr_str(a->session->up_peer), a->session->up_seid);
+	OSMO_STRBUF_PRINTF(sb, " PDR-access:%d", a->pdr_access);
+	OSMO_STRBUF_PRINTF(sb, " PDR-core:%d", a->pdr_core);
 	return sb.chars_needed;
 }
 

src/osmo-upf/up_peer.c

@@ -169,11 +169,6 @@ struct up_peer *up_peer_find_or_add(struct up_endpoint *up_endpoint, const struc
 	return up_peer_add(up_endpoint, remote_addr);
 }
 
-int up_peer_tx(struct up_peer *peer, struct osmo_pfcp_msg *m)
-{
-	return osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, m);
-}
-
 static int up_peer_fsm_timer_cb(struct osmo_fsm_inst *fi)
 {
 	//struct up_peer *peer = fi->priv;
@@ -188,7 +183,7 @@ void up_peer_set_msg_ctx(struct up_peer *peer, struct osmo_pfcp_msg *m)
 	m->ctx.peer_fi = peer->fi;
 	m->ctx.peer_use_count = &peer->use_count;
 	m->ctx.peer_use_token = (m->rx ? UP_USE_MSG_RX : UP_USE_MSG_TX);
-	osmo_use_count_get_put(m->ctx.peer_use_count, m->ctx.peer_use_token, 1);
+	OSMO_ASSERT(osmo_use_count_get_put(m->ctx.peer_use_count, m->ctx.peer_use_token, 1) == 0);
 }
 
 struct osmo_pfcp_msg *up_peer_init_tx(struct up_peer *peer, struct osmo_pfcp_msg *in_reply_to,
@@ -217,7 +212,8 @@ static int up_peer_tx_assoc_setup_resp(struct up_peer *peer, struct osmo_pfcp_ms
 	};
 
 	if (osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, resp)) {
-		OSMO_LOG_PFCP_MSG(resp, LOGL_ERROR, "Error sending response, cannot associate with peer\n");
+		OSMO_LOG_PFCP_MSG(m, LOGL_ERROR, "Error sending response to this message,"
+				  " cannot associate with peer\n");
 		return -EIO;
 	}
 	return 0;
@@ -234,7 +230,7 @@ static int up_peer_tx_assoc_rel_resp(struct up_peer *peer, struct osmo_pfcp_msg
 	};
 
 	if (osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, resp)) {
-		OSMO_LOG_PFCP_MSG(resp, LOGL_ERROR, "Error sending response\n");
+		OSMO_LOG_PFCP_MSG(m, LOGL_ERROR, "Error sending response to this message\n");
 		return -EIO;
 	}
 	return 0;
@@ -303,7 +299,7 @@ static void up_peer_rx_session_est_req(struct up_peer *peer, struct osmo_pfcp_ms
 {
 	enum osmo_pfcp_cause cause = OSMO_PFCP_CAUSE_REQUEST_ACCEPTED;
 	struct osmo_pfcp_msg *resp;
-	struct up_session *session = up_session_find_or_add(peer, &m->ies.session_est_req.cp_f_seid, NULL);
+	struct up_session *session = up_session_find_or_add(peer, &m->ies.session_est_req.cp_f_seid);
 
 	if (!session) {
 		cause = OSMO_PFCP_CAUSE_NO_RESOURCES_AVAILABLE;
@@ -326,6 +322,8 @@ nack_response:
 		.cause = cause,
 	};
 	osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, resp);
+	if (session)
+		up_session_discard(session);
 }
 
 static void up_peer_not_associated_action(struct osmo_fsm_inst *fi, uint32_t event, void *data)

src/osmo-upf/up_session.c

@@ -34,6 +34,7 @@
 #include <osmocom/upf/up_peer.h>
 #include <osmocom/upf/up_session.h>
 #include <osmocom/upf/up_gtp_action.h>
+#include <osmocom/upf/netinst.h>
 
 static enum osmo_pfcp_cause up_session_setup_gtp(struct up_session *session);
 
@@ -47,7 +48,7 @@ void up_session_set_msg_ctx(struct up_session *session, struct osmo_pfcp_msg *m)
 	m->ctx.session_fi = session->fi;
 	m->ctx.session_use_count = &session->use_count;
 	m->ctx.session_use_token = (m->rx ? UP_USE_MSG_RX : UP_USE_MSG_TX);
-	osmo_use_count_get_put(m->ctx.session_use_count, m->ctx.session_use_token, 1);
+	OSMO_ASSERT(osmo_use_count_get_put(m->ctx.session_use_count, m->ctx.session_use_token, 1) == 0);
 }
 
 enum up_session_fsm_state {
@@ -114,13 +115,76 @@ struct chosen_f_teid *chosen_f_teid_find(struct llist_head *list, uint8_t choose
 	return NULL;
 }
 
+/* Find local interface's IP address by Network Instance name. Return 0 on success, or an OSMO_PFCP_CAUSE_* value on
+ * failure. */
+static int up_session_choose_local_ip(struct up_session *session, struct osmo_pfcp_ip_addrs *local_addr,
+				      const char *netinst_name)
+{
+	const struct network_instance *netinst;
+	struct osmo_sockaddr osa = {};
+
+	if (llist_empty(&g_upf->netinst)) {
+		/* No network instances are configured in osmo-upf.cfg. Instead use the local address configured for
+		 * PFCP, assuming that in a simplistic setup the host has only one interface. It is unlikely to be
+		 * useful for a production environment where the entire point is to hand packet data from one interface
+		 * to another, and where PFCP most probably happens on an entirely different interface, but may make
+		 * things simpler for lab testing. */
+		if (osmo_pfcp_ip_addrs_set(local_addr,
+					   osmo_pfcp_endpoint_get_local_addr(session->up_peer->up_endpoint->pfcp_ep))) {
+			LOGPFSML(session->fi, LOGL_ERROR, "Invalid local address in pfcp_endpoint cfg\n");
+			return OSMO_PFCP_CAUSE_SYSTEM_FAILURE;
+		}
+		LOGPFSML(session->fi, LOGL_NOTICE,
+			 "Cannot look up Network Instance %s: No 'netinst' is configured, setting up GTP on same local"
+			 " interface as PFCP: %s (makes sense only for lab testing)\n",
+			 osmo_quote_str_c(OTC_SELECT, netinst_name, -1),
+			 osmo_pfcp_ip_addrs_to_str_c(OTC_SELECT, local_addr));
+		return 0;
+	}
+
+	if (!netinst_name || !*netinst_name) {
+		/* Empty or no Network Instance IE in incoming PFCP request. Pick the first network instance; makes
+		 * sense only in a simplistic lab setup where packet data is forwarded to the same interface that it is
+		 * received on, and where no Network Instance is indicated by the CPF. Warn if more than one network
+		 * instance is configured to choose from. */
+		if (llist_count(&g_upf->netinst) > 1)
+			LOGPFSML(session->fi, LOGL_NOTICE,
+				 "Missing Network Instance in incoming request, using the first 'netinst' from cfg\n");
+		netinst = netinst_first(&g_upf->netinst);
+		/* there has to be a first entry, because we handled the empty list above. */
+		OSMO_ASSERT(netinst);
+	} else {
+		netinst = netinst_find(&g_upf->netinst, netinst_name);
+		if (!netinst) {
+			LOGPFSML(session->fi, LOGL_ERROR, "Network Instance from PFCP request not found: %s"
+				 " -- ensure there is a 'netinst' / 'add %s <ip-addr>' entry in your config\n",
+				 osmo_quote_str_c(OTC_SELECT, netinst_name, -1),
+				 osmo_escape_str_c(OTC_SELECT, netinst_name, -1));
+			return OSMO_PFCP_CAUSE_RULE_CREATION_MOD_FAILURE;
+		}
+	}
+
+	/* Convert netinst IP address string first to osmo_sockaddr and then to osmo_pfcp_ip_addrs. */
+	if (osmo_sockaddr_str_to_sockaddr(&netinst->addr, &osa.u.sas)
+	    || osmo_pfcp_ip_addrs_set(local_addr, &osa)) {
+		LOGPFSML(session->fi, LOGL_ERROR,
+			 "Network Instance %s from PFCP request yields no valid IP address: "
+				OSMO_SOCKADDR_STR_FMT "\n",
+			 osmo_quote_str_c(OTC_SELECT, netinst_name, -1),
+			 OSMO_SOCKADDR_STR_FMT_ARGS(&netinst->addr));
+		return OSMO_PFCP_CAUSE_RULE_CREATION_MOD_FAILURE;
+	}
+	return 0;
+}
+
 /* Choose an F-TEID (when the peer has sent CHOOSE = 1).
  * If the peer also sent a CHOOSE_ID, then remember this F-TEID choice under the given ID, and re-use that choice when
- * the same ID re-appears. The chosen IDs are saved in session->chosen_f_teids. */
+ * the same ID re-appears. The chosen IDs are saved in session->chosen_f_teids.
+ * Return 0 on success, or an OSMO_PFCP_CAUSE_* value on failure. */
 static enum osmo_pfcp_cause up_session_choose_f_teid(struct up_session *session, struct osmo_pfcp_ie_f_teid *dst,
-						     bool choose_id_present, uint8_t choose_id)
+						     bool choose_id_present, uint8_t choose_id,
+						     const char *netinst_name)
 {
-	struct up_endpoint *up_ep = session->up_peer->up_endpoint;
 	struct chosen_f_teid *chosen = NULL;
 
 	if (choose_id_present)
@@ -129,23 +193,26 @@ static enum osmo_pfcp_cause up_session_choose_f_teid(struct up_session *session,
 		/* Re-use a previous F-TEID */
 		*dst = chosen->f_teid;
 	} else {
-		/* Choose a new F-TEID */
+		int rc;
+
 		*dst = (struct osmo_pfcp_ie_f_teid){
-			.fixed = {
-				.teid = up_endpoint_next_teid(up_ep),
-			},
+			.choose_flag = false,
 		};
+
+		/* Determine local IP address from Network Instance value received in PFCP request */
+		rc = up_session_choose_local_ip(session, &dst->fixed.ip_addr, netinst_name);
+		if (rc)
+			return rc;
+
+		/* Choose a new TEID */
+		dst->fixed.teid = upf_next_local_teid();
 		if (dst->fixed.teid == 0) {
 			LOGPFSML(session->fi, LOGL_ERROR, "Failed to allocate an unused TEID\n");
 			return OSMO_PFCP_CAUSE_PFCP_ENTITY_IN_CONGESTION;
 		}
-		LOGPFSML(session->fi, LOGL_INFO, "Allocated new local TEID 0x%x\n", dst->fixed.teid);
+		LOGPFSML(session->fi, LOGL_INFO, "Allocated new local F-TEID %s\n",
+			 osmo_pfcp_ie_f_teid_to_str_c(OTC_SELECT, dst));
 
-		if (osmo_pfcp_ip_addrs_set(&dst->fixed.ip_addr,
-					   osmo_pfcp_endpoint_get_local_addr(up_ep->pfcp_ep))) {
-			LOGPFSML(session->fi, LOGL_ERROR, "Invalid local address in pfcp_endpoint cfg\n");
-			return OSMO_PFCP_CAUSE_PFCP_ENTITY_IN_CONGESTION;
-		}
 		/* Save this choice */
 		if (choose_id_present) {
 			chosen = talloc(session, struct chosen_f_teid);
@@ -189,22 +256,23 @@ static void far_upd(struct far *far, const struct osmo_pfcp_ie_upd_far *upd)
 	if (upd->upd_forw_params_present) {
 		const struct osmo_pfcp_ie_upd_forw_params *u = &upd->upd_forw_params;
 		struct osmo_pfcp_ie_forw_params *p = &far->desc.forw_params;
+		far->desc.forw_params_present = true;
 		if (u->destination_iface_present)
 			p->destination_iface = u->destination_iface;
 		if (u->network_inst_present) {
-			p->network_inst = p->network_inst;
+			p->network_inst = u->network_inst;
 			p->network_inst_present = true;
 		}
 		if (u->outer_header_creation_present) {
-			p->outer_header_creation = p->outer_header_creation;
+			p->outer_header_creation = u->outer_header_creation;
 			p->outer_header_creation_present = true;
 		}
 		if (u->linked_te_id_present) {
-			p->linked_te_id = p->linked_te_id;
+			p->linked_te_id = u->linked_te_id;
 			p->linked_te_id_present = true;
 		}
 		if (u->destination_iface_type_present) {
-			p->destination_iface_type = p->destination_iface_type;
+			p->destination_iface_type = u->destination_iface_type;
 			p->destination_iface_type_present = true;
 		}
 	}
@@ -237,12 +305,9 @@ static int far_to_str_buf(char *buf, size_t len, const struct far *far)
 	if (f->forw_params_present) {
 		OSMO_STRBUF_PRINTF(sb, " dst:%s", osmo_pfcp_dest_iface_str(f->forw_params.destination_iface));
 		if (f->forw_params.outer_header_creation_present) {
-			OSMO_STRBUF_PRINTF(sb, " encaps-");
-			OSMO_STRBUF_APPEND(sb, osmo_pfcp_bits_to_str_buf,
-					   f->forw_params.outer_header_creation.desc_bits,
-					   osmo_pfcp_outer_header_creation_strs);
-			if (f->forw_params.outer_header_creation.teid_present)
-				OSMO_STRBUF_PRINTF(sb, " TEID-0x%x", f->forw_params.outer_header_creation.teid);
+			OSMO_STRBUF_PRINTF(sb, ",");
+			OSMO_STRBUF_APPEND(sb, osmo_pfcp_ie_outer_header_creation_to_str_buf,
+					   &f->forw_params.outer_header_creation);
 		}
 	}
 	OSMO_STRBUF_PRINTF(sb, "}");
@@ -273,6 +338,10 @@ int pdr_to_str_buf(char *buf, size_t buflen, const struct pdr *pdr)
 			OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &pdr->desc.pdi.ue_ip_address.ip_addr.v6);
 		}
 	}
+	if (pdr->desc.pdi.network_inst_present) {
+		OSMO_STRBUF_PRINTF(sb, " netinst:");
+		OSMO_STRBUF_APPEND(sb, osmo_quote_str_buf3, pdr->desc.pdi.network_inst.str, -1);
+	}
 	if (pdr->local_f_teid) {
 		OSMO_STRBUF_PRINTF(sb, " ");
 		OSMO_STRBUF_APPEND(sb, osmo_pfcp_ie_f_teid_to_str_buf, pdr->local_f_teid);
@@ -316,6 +385,7 @@ static void pdr_set_far(struct pdr *pdr, struct far *far)
 	pdr->far = far;
 }
 
+/* Set up a new Packet Detection Rule, append the response to the end of the created_pdr/created_pdr_count array. */
 static struct pdr *pdr_create(struct up_session *session,
 			      const struct osmo_pfcp_ie_create_pdr *create_pdr,
 			      enum osmo_pfcp_cause *cause,
@@ -372,9 +442,13 @@ static struct pdr *pdr_create(struct up_session *session,
 		if (pdr->desc.pdi.local_f_teid.choose_flag) {
 			/* CHOOSE = 1: we need to pick our own local F-TEID */
 			struct osmo_pfcp_ie_f_teid local_f_teid;
+			const char *netinst_name = NULL;
+			if (pdr->desc.pdi.network_inst_present)
+				netinst_name = pdr->desc.pdi.network_inst.str;
 			*cause = up_session_choose_f_teid(session, &local_f_teid,
 							  pdr->desc.pdi.local_f_teid.choose.choose_id_present,
-							  pdr->desc.pdi.local_f_teid.choose.choose_id);
+							  pdr->desc.pdi.local_f_teid.choose.choose_id,
+							  netinst_name);
 			if (*cause != OSMO_PFCP_CAUSE_REQUEST_ACCEPTED) {
 				*offending_ie = OSMO_PFCP_IEI_F_TEID;
 				*offending_ie_present = true;
@@ -482,8 +556,7 @@ static struct pdr *pdr_upd(struct pdr *pdr,
 	return pdr;
 
 nack_resp:
-	if (pdr)
-		pdr_del(pdr);
+	pdr_del(pdr);
 	if (!*offending_ie_present) {
 		*offending_ie = OSMO_PFCP_IEI_UPD_PDR;
 		*offending_ie_present = true;
@@ -556,14 +629,20 @@ static void up_session_est(struct up_session *session, struct osmo_pfcp_msg *m)
 	resp->up_f_seid_present = true;
 
 	rc = osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx);
-	if (rc)
+	if (rc) {
+		/* sending ACK failed, discard session. It might seem like a good idea to keep the session around,
+		 * because the creation succeeded, only the ACK failed. But in the greater scheme of things, if we
+		 * cannot ACK to the PFCP peer, all is lost. Rather not keep stale sessions around. */
 		up_session_fsm_state_chg(UP_SESSION_ST_WAIT_USE_COUNT);
+		return;
+	}
 	up_session_fsm_state_chg(UP_SESSION_ST_ESTABLISHED);
 	return;
 
 nack_response:
 	resp->created_pdr_count = 0;
 	osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx);
+	/* No matter if sending the NACK succeeded or not, discard the session. */
 	up_session_fsm_state_chg(UP_SESSION_ST_WAIT_USE_COUNT);
 }
 
@@ -644,8 +723,13 @@ static void up_session_mod(struct up_session *session, struct osmo_pfcp_msg *m)
 		goto nack_response;
 
 	/* Success, send ACK */
-	if (osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx))
+	if (osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx)) {
+		/* sending ACK failed, discard session. It might seem like a good idea to keep the session around,
+		 * because the modification succeeded, only the ACK failed. But in the greater scheme of things, if we
+		 * cannot ACK to the PFCP peer, all is lost. Rather not keep stale sessions around. */
 		up_session_fsm_state_chg(UP_SESSION_ST_WAIT_USE_COUNT);
+		return;
+	}
 
 	LOGPFSML(fi, LOGL_NOTICE, "Session modified: %s\n", up_session_gtp_status(session));
 	return;
@@ -653,6 +737,7 @@ static void up_session_mod(struct up_session *session, struct osmo_pfcp_msg *m)
 nack_response:
 	resp->created_pdr_count = 0;
 	osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx);
+	/* No matter if sending the NACK succeeded or not, discard the session. */
 	up_session_fsm_state_chg(UP_SESSION_ST_WAIT_USE_COUNT);
 }
 
@@ -667,6 +752,7 @@ static void up_session_del(struct up_session *session, struct osmo_pfcp_msg *m)
 		.cause = OSMO_PFCP_CAUSE_REQUEST_ACCEPTED
 	};
 	osmo_pfcp_endpoint_tx(peer->up_endpoint->pfcp_ep, tx);
+	/* No matter if sending the deletion ACK succeeded or not, discard the session. */
 	up_session_fsm_state_chg(UP_SESSION_ST_WAIT_USE_COUNT);
 }
 
@@ -899,7 +985,7 @@ static inline uint64_t up_session_key(uint64_t cp_seid, uint64_t up_seid)
 static struct up_session *up_session_add(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid)
 {
 	struct up_session *session;
-	uint64_t up_seid = up_endpoint_next_seid(peer->up_endpoint);
+	uint64_t up_seid = up_endpoint_next_up_seid(peer->up_endpoint);
 
 	if (!up_seid)
 		return NULL;
@@ -934,16 +1020,11 @@ static struct up_session *up_session_add(struct up_peer *peer, const struct osmo
 	return session;
 }
 
-struct up_session *up_session_find_or_add(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid,
-					  const struct osmo_pfcp_ie_f_seid *up_f_seid)
+struct up_session *up_session_find_or_add(struct up_peer *peer, const struct osmo_pfcp_ie_f_seid *cp_f_seid)
 {
 	struct up_session *session;
-	if (cp_f_seid)
-		session = up_session_find_by_cp_f_seid(peer, cp_f_seid);
-	else if (up_f_seid)
-		session = up_session_find_by_up_seid(peer, up_f_seid->seid);
-	else
-		return NULL;
+	OSMO_ASSERT(cp_f_seid);
+	session = up_session_find_by_cp_f_seid(peer, cp_f_seid);
 	if (session)
 		return session;
 
@@ -996,8 +1077,8 @@ static void pdr_classify(struct pdr *pdr)
 {
 	pdr->rx_decaps = false;
 	pdr->forw_encaps = false;
-	pdr->forw_to_core = false;
-	pdr->forw_from_core = false;
+	pdr->access_to_core = false;
+	pdr->core_to_access = false;
 	if (!pdr->far)
 		return;
 
@@ -1009,10 +1090,10 @@ static void pdr_classify(struct pdr *pdr)
 	if (!action_is_forw(&pdr->far->desc.apply_action))
 		return;
 
-	pdr->forw_to_core = (pdr->desc.pdi.source_iface == OSMO_PFCP_SOURCE_IFACE_ACCESS
-			     && pdr->far->desc.forw_params.destination_iface == OSMO_PFCP_DEST_IFACE_CORE);
+	pdr->access_to_core = (pdr->desc.pdi.source_iface == OSMO_PFCP_SOURCE_IFACE_ACCESS
+			       && pdr->far->desc.forw_params.destination_iface == OSMO_PFCP_DEST_IFACE_CORE);
 
-	pdr->forw_from_core = (pdr->desc.pdi.source_iface == OSMO_PFCP_SOURCE_IFACE_CORE
+	pdr->core_to_access = (pdr->desc.pdi.source_iface == OSMO_PFCP_SOURCE_IFACE_CORE
 			       && pdr->far->desc.forw_params.destination_iface == OSMO_PFCP_DEST_IFACE_ACCESS);
 }
 
@@ -1032,6 +1113,11 @@ void pdr_reverse_unset(struct pdr *pdr)
 	pdr->reverse_pdr = NULL;
 }
 
+/* Log that a PDR (and its reverse-PDR) is inactive.
+ * \param pdr  The Access-to-Core PDR.
+ * \param desc  Why it is inactive.
+ * \param pdr_to_str  The PDR that desc describes, can be pdr or the reverse Core-to-Access PDR.
+ */
 static void log_inactive_pdr_set(struct pdr *pdr, const char *desc, const struct pdr *pdr_to_str)
 {
 	struct pdr *rpdr = pdr->reverse_pdr;
@@ -1055,8 +1141,9 @@ static void log_inactive_pdr_set(struct pdr *pdr, const char *desc, const struct
  * The given PDR must have an outer-header-removal and a local F-TEID.
  * Its reverse-PDR must have a UE address flagged as "Destination" IP addr.
  * Its reverse-PDR's FAR must have an outer-header creation with a remote TEID.
+ * \param pdr  A rule detecting packets on Access, where pdr->reverse_pdr detects packets on Core.
  */
-static void add_gtp_action_endecaps(void *ctx, struct llist_head *dst, struct pdr *pdr)
+static void add_gtp_action_tunend(void *ctx, struct llist_head *dst, struct pdr *pdr)
 {
 	struct up_session *session = pdr->session;
 	struct up_gtp_action *a;
@@ -1067,18 +1154,20 @@ static void add_gtp_action_endecaps(void *ctx, struct llist_head *dst, struct pd
 	OSMO_ASSERT(pdr->far);
 	OSMO_ASSERT(pdr->reverse_pdr);
 	OSMO_ASSERT(pdr->reverse_pdr->far);
-
-	/* To decaps, we need to have a local TEID assigned for which to receive GTP packets. */
-	if (!pdr->local_f_teid || pdr->local_f_teid->choose_flag) {
-		log_inactive_pdr_set(pdr, "missing local TEID", pdr);
-		return;
-	}
-
-	/* To encaps, we need to have a remote TEID assigned to send out in GTP packets, and we need to know where to
-	 * send GTP to. */
 	rpdr = pdr->reverse_pdr;
 	rfar = rpdr->far;
 	rfar_forw = &rfar->desc.forw_params;
+
+	OSMO_ASSERT(pdr->access_to_core);
+	OSMO_ASSERT(rpdr->core_to_access);
+
+	/* To decaps incoming on Access, we need to have a local F-TEID assigned for which to receive GTP packets. */
+	if (!pdr->local_f_teid || pdr->local_f_teid->choose_flag) {
+		log_inactive_pdr_set(pdr, "missing local F-TEID", pdr);
+		return;
+	}
+
+	/* To encaps outgoing on Access, we need to have a remote F-TEID assigned to send out in GTP packets */
 	if (!rfar->desc.forw_params_present) {
 		log_inactive_pdr_set(pdr, "missing FAR Forwarding Parameters", rpdr);
 		return;
@@ -1096,8 +1185,7 @@ static void add_gtp_action_endecaps(void *ctx, struct llist_head *dst, struct pd
 		return;
 	}
 
-	/* To receive packets to be encapsulated, we need to know the assigned IP address for the UE, which receives the
-	 * IP packets that should be placed into GTP. */
+	/* To receive IP packets incoming on Core, we need to know the assigned IP address for the UE */
 	if (!rpdr->desc.pdi.ue_ip_address_present) {
 		log_inactive_pdr_set(pdr, "missing UE IP Address in PDI", rpdr);
 		return;
@@ -1127,21 +1215,35 @@ static void add_gtp_action_endecaps(void *ctx, struct llist_head *dst, struct pd
 	OSMO_ASSERT(a);
 	*a = (struct up_gtp_action){
 		.session = session,
-		.pdr_core = pdr->desc.pdr_id,
-		.pdr_access = rpdr->desc.pdr_id,
-		.kind = UP_GTP_U_ENDECAPS,
-		.endecaps = {
-			.local_teid = pdr->local_f_teid->fixed.teid,
-			.remote_teid = rfar_forw->outer_header_creation.teid,
-			.gtp_remote_addr = rfar_forw->outer_header_creation.ip_addr.v4,
-			.ue_addr = rpdr->desc.pdi.ue_ip_address.ip_addr.v4,
+		.pdr_access = pdr->desc.pdr_id,
+		.pdr_core = rpdr->desc.pdr_id,
+		.kind = UP_GTP_U_TUNEND,
+		.tunend = {
+			.access = {
+				.local = {
+					.addr = pdr->local_f_teid->fixed.ip_addr.v4,
+					.teid = pdr->local_f_teid->fixed.teid,
+				},
+				.remote = {
+					.addr = rfar_forw->outer_header_creation.ip_addr.v4,
+					.teid = rfar_forw->outer_header_creation.teid,
+				},
+			},
+			.core = {
+				.ue_local_addr = rpdr->desc.pdi.ue_ip_address.ip_addr.v4,
+			},
 		},
 	};
 
 	llist_add_tail(&a->entry, dst);
 }
 
-static void add_gtp_action_forw(void *ctx, struct llist_head *dst, struct pdr *pdr)
+/* A GTP tunnel on Access side, mapping to another GTP tunnel on Core side and vice versa.
+ * The PDR and its reverse PDR must both have an outer-header-removal and a local F-TEID.
+ * Both FARs must have an outer-header creation with a remote F-TEID.
+ * \param pdr  A rule detecting packets on Access, where pdr->reverse_pdr detects packets on Core.
+ */
+static void add_gtp_action_tunmap(void *ctx, struct llist_head *dst, struct pdr *pdr)
 {
 	struct up_session *session = pdr->session;
 	struct up_gtp_action *a;
@@ -1161,52 +1263,53 @@ static void add_gtp_action_forw(void *ctx, struct llist_head *dst, struct pdr *p
 	rfar = rpdr->far;
 	rfar_forw = &rfar->desc.forw_params;
 
-	/* To decaps, we need to have a local TEID assigned for which to receive GTP packets. */
-	/* decaps from CORE */
+	OSMO_ASSERT(pdr->access_to_core);
+	OSMO_ASSERT(rpdr->core_to_access);
+
+	/* To decaps incoming on Access, we need to have a local F-TEID assigned for which to receive GTP packets. */
 	if (!pdr->local_f_teid || pdr->local_f_teid->choose_flag) {
-		log_inactive_pdr_set(pdr, "missing local TEID (CORE side)", pdr);
+		log_inactive_pdr_set(pdr, "missing local F-TEID (Access side)", pdr);
 		return;
 	}
-	/* decaps from ACCESS */
+	/* To decaps incoming on Core, we need to have a local F-TEID assigned for which to receive GTP packets. */
 	if (!rpdr->local_f_teid || rpdr->local_f_teid->choose_flag) {
-		log_inactive_pdr_set(pdr, "missing local TEID (ACCESS side)", pdr);
+		log_inactive_pdr_set(pdr, "missing local F-TEID (Core side)", pdr);
 		return;
 	}
 
-	/* To encaps, we need to have a remote TEID assigned to send out in GTP packets, and we need to know where to
-	 * send GTP to. */
-	/* encaps towards ACCESS */
+	/* To encaps outgoing on Core, we need to have a remote F-TEID assigned to send out in GTP packets */
 	if (!far->desc.forw_params_present) {
-		log_inactive_pdr_set(pdr, "missing FAR Forwarding Parameters", pdr);
+		log_inactive_pdr_set(pdr, "missing FAR Forwarding Parameters (Access side)", pdr);
 		return;
 	}
 	if (!far_forw->outer_header_creation_present) {
-		log_inactive_pdr_set(pdr, "missing FAR Outer Header Creation", pdr);
+		log_inactive_pdr_set(pdr, "missing FAR Outer Header Creation (Access side)", pdr);
 		return;
 	}
 	if (!far_forw->outer_header_creation.teid_present) {
-		log_inactive_pdr_set(pdr, "missing TEID in FAR Outer Header Creation", pdr);
+		log_inactive_pdr_set(pdr, "missing TEID in FAR Outer Header Creation (Access side)", pdr);
 		return;
 	}
 	if (!far_forw->outer_header_creation.ip_addr.v4_present) {
-		log_inactive_pdr_set(pdr, "missing IPv4 in FAR Outer Header Creation", pdr);
+		log_inactive_pdr_set(pdr, "missing IPv4 in FAR Outer Header Creation (Access side)", pdr);
 		return;
 	}
-	/* encaps towards CORE */
+
+	/* To encaps outgoing on Access, we need to have a remote F-TEID assigned to send out in GTP packets */
 	if (!rfar->desc.forw_params_present) {
-		log_inactive_pdr_set(pdr, "missing FAR Forwarding Parameters", rpdr);
+		log_inactive_pdr_set(pdr, "missing FAR Forwarding Parameters (Access side)", rpdr);
 		return;
 	}
 	if (!rfar_forw->outer_header_creation_present) {
-		log_inactive_pdr_set(pdr, "missing FAR Outer Header Creation", rpdr);
+		log_inactive_pdr_set(pdr, "missing FAR Outer Header Creation (Access side)", rpdr);
 		return;
 	}
 	if (!rfar_forw->outer_header_creation.teid_present) {
-		log_inactive_pdr_set(pdr, "missing TEID in FAR Outer Header Creation", rpdr);
+		log_inactive_pdr_set(pdr, "missing TEID in FAR Outer Header Creation (Access side)", rpdr);
 		return;
 	}
 	if (!rfar_forw->outer_header_creation.ip_addr.v4_present) {
-		log_inactive_pdr_set(pdr, "missing IPv4 in FAR Outer Header Creation", rpdr);
+		log_inactive_pdr_set(pdr, "missing IPv4 in FAR Outer Header Creation (Access side)", rpdr);
 		return;
 	}
 
@@ -1226,19 +1329,29 @@ static void add_gtp_action_forw(void *ctx, struct llist_head *dst, struct pdr *p
 	OSMO_ASSERT(a);
 	*a = (struct up_gtp_action){
 		.session = session,
-		.pdr_core = pdr->desc.pdr_id,
-		.pdr_access = rpdr->desc.pdr_id,
+		.pdr_access = pdr->desc.pdr_id,
+		.pdr_core = rpdr->desc.pdr_id,
 		.kind = UP_GTP_U_TUNMAP,
 		.tunmap = {
-			.core = {
-				.local_teid = pdr->local_f_teid->fixed.teid,
-				.remote_teid = rfar_forw->outer_header_creation.teid,
-				.gtp_remote_addr = rfar_forw->outer_header_creation.ip_addr.v4,
+			.access.tun = {
+				.local = {
+					.addr = pdr->local_f_teid->fixed.ip_addr.v4,
+					.teid = pdr->local_f_teid->fixed.teid,
+				},
+				.remote = {
+					.addr = rfar_forw->outer_header_creation.ip_addr.v4,
+					.teid = rfar_forw->outer_header_creation.teid,
+				},
 			},
-			.access = {
-				.local_teid = rpdr->local_f_teid->fixed.teid,
-				.remote_teid = far_forw->outer_header_creation.teid,
-				.gtp_remote_addr = far_forw->outer_header_creation.ip_addr.v4,
+			.core.tun = {
+				.local =  {
+					.addr = rpdr->local_f_teid->fixed.ip_addr.v4,
+					.teid = rpdr->local_f_teid->fixed.teid,
+				},
+				.remote = {
+					.addr = far_forw->outer_header_creation.ip_addr.v4,
+					.teid = far_forw->outer_header_creation.teid,
+				},
 			},
 		},
 	};
@@ -1270,12 +1383,13 @@ static enum osmo_pfcp_cause find_gtp_actions(void *ctx, struct llist_head *dst,
 		if (pdr->reverse_pdr)
 			continue;
 
-		/* In this outer loop, only follow the forw_to_core directed PDRs, in the inner loop find the matching
-		 * forw_from_core PDR. */
-		if (!pdr->forw_to_core)
+		/* In this outer loop, only follow the access_to_core directed PDRs, in the inner loop find the matching
+		 * core_to_access PDR. i.e. we are looking only at PDRs detecting packets on the Access side, pairing up
+		 * with "reverse PDRs" detecting packets on the Core side. */
+		if (!pdr->access_to_core)
 			continue;
 
-		/* If a required TEID is not known, we cannot pair this PDR up */
+		/* If a required local addr + TEID is not known, we cannot pair this PDR up */
 		if (pdr->rx_decaps && !pdr->local_f_teid)
 			continue;
 
@@ -1286,7 +1400,7 @@ static enum osmo_pfcp_cause find_gtp_actions(void *ctx, struct llist_head *dst,
 				continue;
 
 			/* Looking for a PDR facing the other way */
-			if (!other->forw_from_core)
+			if (!other->core_to_access)
 				continue;
 			/* GTP header-ness must match, in reverse. */
 			if (pdr->rx_decaps != other->forw_encaps
@@ -1309,14 +1423,14 @@ static enum osmo_pfcp_cause find_gtp_actions(void *ctx, struct llist_head *dst,
 			continue;
 		}
 
-		/* Iterate in direction to-Core, where pdr->reverse_pdr will be the from-Core counterpart. */
-		if (!pdr->forw_to_core)
+		/* Iterate in direction Access-to-Core, where pdr->reverse_pdr will be the Core-to-Access counterpart. */
+		if (!pdr->access_to_core)
 			continue;
 
 		if (pdr->rx_decaps && !pdr->forw_encaps)
-			add_gtp_action_endecaps(ctx, dst, pdr);
+			add_gtp_action_tunend(ctx, dst, pdr);
 		else if (pdr->rx_decaps && pdr->forw_encaps)
-			add_gtp_action_forw(ctx, dst, pdr);
+			add_gtp_action_tunmap(ctx, dst, pdr);
 		else {
 			/* log the details of both PDRs in two separate log lines */
 			log_inactive_pdr_set(pdr, "not implemented", pdr);
@@ -1414,11 +1528,16 @@ static enum osmo_pfcp_cause up_session_setup_gtp(struct up_session *session)
 	return cause;
 }
 
+/* Return true when the session is in Established state and has active GTP actions. */
 bool up_session_is_active(struct up_session *session)
 {
 	return session && (session->fi->state == UP_SESSION_ST_ESTABLISHED) && !llist_empty(&session->active_gtp_actions);
 }
 
+/* Return true when up_session_is_active() == true *and* it has only active PDR/FAR pairs.
+ * A PDR/FAR is inactive when it is not part of an active GTP action. Reasons may be that it has no PDR-to-FAR relation,
+ * there is no matching reverse PDR/FAR, that a FAR is not set to FORW, an ignored Source/Destination Interface, ...
+ */
 bool up_session_is_fully_active(struct up_session *session, int *active_p, int *inactive_p)
 {
 	struct pdr *pdr;

src/osmo-upf/upf.c

@@ -29,12 +29,27 @@
 
 #include <osmocom/upf/upf.h>
 #include <osmocom/upf/up_endpoint.h>
+#include <osmocom/upf/up_peer.h>
+#include <osmocom/upf/up_session.h>
+#include <osmocom/upf/up_gtp_action.h>
 #include <osmocom/upf/upf_gtp.h>
 
 struct g_upf *g_upf = NULL;
 
+struct osmo_tdef g_upf_nft_tdefs[] = {
+	{ .T = -32, .default_val = 1000, .unit = OSMO_TDEF_MS,
+	  .desc = "How long to wait for more nft rulesets before flushing in batch",
+	},
+	{ .T = -33, .default_val = 1, .unit = OSMO_TDEF_CUSTOM,
+	  .desc = "When reaching this nr of queued nft rulesets, flush the queue",
+	  .max_val = 128,
+	},
+	{}
+};
+
 struct osmo_tdef_group g_upf_tdef_groups[] = {
 	{ "pfcp", "PFCP endpoint timers", osmo_pfcp_tdefs, },
+	{ "nft", "netfilter timers", g_upf_nft_tdefs, },
 	{}
 };
 
@@ -50,20 +65,22 @@ void g_upf_alloc(void *ctx)
 				.local_port = OSMO_PFCP_PORT,
 			},
 		},
-		.nft = {
-			.priority = -300,
+		.tunmap = {
+			.priority_pre = -300,
+			.priority_post = 400,
 		},
-		.gtp = {
+		.tunend = {
 			/* TODO: recovery count state file; use lower byte of current time, poor person's random. */
 			.recovery_count = time(NULL),
 		},
 	};
 
-	INIT_LLIST_HEAD(&g_upf->gtp.vty_cfg.devs);
-	INIT_LLIST_HEAD(&g_upf->gtp.devs);
+	INIT_LLIST_HEAD(&g_upf->tunend.vty_cfg.devs);
+	INIT_LLIST_HEAD(&g_upf->tunend.devs);
+	INIT_LLIST_HEAD(&g_upf->netinst);
 }
 
-int upf_pfcp_listen()
+int upf_pfcp_init(void)
 {
 	struct osmo_sockaddr_str local_addr_str;
 	struct osmo_sockaddr local_addr;
@@ -75,9 +92,8 @@ int upf_pfcp_listen()
 	 * osmo_sockaddr. */
 	osmo_sockaddr_str_from_str(&local_addr_str, g_upf->pfcp.vty_cfg.local_addr, g_upf->pfcp.vty_cfg.local_port);
 	osmo_sockaddr_str_to_sockaddr(&local_addr_str, &local_addr.u.sas);
-	LOGP(DLPFCP, LOGL_NOTICE, "PFCP: Listening on %s\n", osmo_sockaddr_to_str_c(OTC_SELECT, &local_addr));
 
-	g_upf->pfcp.ep = up_endpoint_init(g_upf, &local_addr);
+	g_upf->pfcp.ep = up_endpoint_alloc(g_upf, &local_addr);
 	if (!g_upf->pfcp.ep) {
 		fprintf(stderr, "Failed to allocate PFCP endpoint.\n");
 		return -1;
@@ -85,10 +101,30 @@ int upf_pfcp_listen()
 	return 0;
 }
 
+int upf_pfcp_listen(void)
+{
+	int rc;
+	if (!g_upf->pfcp.ep) {
+		rc = upf_pfcp_init();
+		if (rc)
+			return rc;
+	}
+
+	rc = up_endpoint_bind(g_upf->pfcp.ep);
+	if (rc) {
+		LOGP(DLPFCP, LOGL_ERROR, "PFCP: failed to listen on %s\n",
+		     osmo_sockaddr_to_str_c(OTC_SELECT, osmo_pfcp_endpoint_get_local_addr(g_upf->pfcp.ep->pfcp_ep)));
+		return rc;
+	}
+	LOGP(DLPFCP, LOGL_NOTICE, "PFCP: Listening on %s\n",
+	     osmo_sockaddr_to_str_c(OTC_SELECT, osmo_pfcp_endpoint_get_local_addr(g_upf->pfcp.ep->pfcp_ep)));
+	return 0;
+}
+
 int upf_gtp_devs_open()
 {
-	struct gtp_vty_cfg *c = &g_upf->gtp.vty_cfg;
-	struct gtp_vty_cfg_dev *d;
+	struct tunend_vty_cfg *c = &g_upf->tunend.vty_cfg;
+	struct tunend_vty_cfg_dev *d;
 
 	llist_for_each_entry(d, &c->devs, entry) {
 		if (upf_gtp_dev_open(d->dev_name, d->create, d->local_addr, false, false))
@@ -96,3 +132,85 @@ int upf_gtp_devs_open()
 	}
 	return 0;
 }
+
+static bool upf_is_local_teid_in_use(uint32_t teid)
+{
+	struct up_peer *peer;
+	llist_for_each_entry(peer, &g_upf->pfcp.ep->peers, entry) {
+		struct up_session *session = up_session_find_by_local_teid(peer, teid);
+		if (session)
+			return true;
+	}
+	return false;
+}
+
+static uint32_t upf_next_local_teid_inc(void)
+{
+	g_upf->gtp.next_local_teid_state++;
+	if (!g_upf->gtp.next_local_teid_state)
+		g_upf->gtp.next_local_teid_state++;
+	return g_upf->gtp.next_local_teid_state;
+}
+
+uint32_t upf_next_local_teid(void)
+{
+	uint32_t sanity;
+	for (sanity = 2342; sanity; sanity--) {
+		uint32_t next_teid = upf_next_local_teid_inc();
+		if (upf_is_local_teid_in_use(next_teid))
+			continue;
+		return next_teid;
+	}
+	return 0;
+}
+
+static uint32_t upf_next_chain_id_inc(void)
+{
+	g_upf->tunmap.next_chain_id_state++;
+	if (!g_upf->tunmap.next_chain_id_state)
+		g_upf->tunmap.next_chain_id_state++;
+	return g_upf->tunmap.next_chain_id_state;
+}
+
+/* Return an unused chain_id, or 0 if none is found with sane effort. */
+uint32_t upf_next_chain_id(void)
+{
+	uint32_t sanity;
+
+	/* Make sure the new chain_id is not used anywhere */
+	for (sanity = 2342; sanity; sanity--) {
+		struct up_peer *peer;
+		uint32_t chain_id = upf_next_chain_id_inc();
+		bool taken = false;
+
+		if (!g_upf->pfcp.ep)
+			return chain_id;
+
+		llist_for_each_entry(peer, &g_upf->pfcp.ep->peers, entry) {
+			struct up_session *session;
+			int bkt;
+			hash_for_each(peer->sessions_by_up_seid, bkt, session, node_by_up_seid) {
+				struct up_gtp_action *a;
+				llist_for_each_entry(a, &session->active_gtp_actions, entry) {
+					if (a->kind != UP_GTP_U_TUNMAP)
+						continue;
+					if (a->tunmap.access.chain_id == chain_id
+					    || a->tunmap.core.chain_id == chain_id) {
+						taken = true;
+						break;
+					}
+				}
+				if (taken)
+					break;
+			}
+			if (taken)
+				break;
+		}
+
+		if (!taken)
+			return chain_id;
+	}
+
+	/* finding a chain_id became insane, return invalid = 0 */
+	return 0;
+}

src/osmo-upf/upf_gtp.c

@@ -39,7 +39,7 @@
 #include <osmocom/upf/upf_gtpu_echo.h>
 
 #define LOG_GTP_TUN(TUN, LEVEL, FMT, ARGS...) \
-	LOGP(DGTP, LEVEL, "%s: " FMT, upf_gtp_tun_to_str_c(OTC_SELECT, (TUN)), ##ARGS)
+	LOGP(DGTP, LEVEL, "%s: " FMT, upf_gtp_tunend_to_str_c(OTC_SELECT, (TUN)), ##ARGS)
 
 int upf_gtp_dev_to_str_buf(char *buf, size_t buflen, const struct upf_gtp_dev *dev)
 {
@@ -66,16 +66,36 @@ char *upf_gtp_dev_to_str_c(void *ctx, const struct upf_gtp_dev *dev)
 struct upf_gtp_dev *upf_gtp_dev_find_by_name(const char *name)
 {
 	struct upf_gtp_dev *dev;
-	llist_for_each_entry(dev, &g_upf->gtp.devs, entry) {
+	llist_for_each_entry(dev, &g_upf->tunend.devs, entry) {
 		if (!strcmp(name, dev->name))
 			return dev;
 	}
 	return NULL;
 }
 
+struct upf_gtp_dev *upf_gtp_dev_find_by_local_addr(const struct osmo_sockaddr *local_addr)
+{
+	struct upf_gtp_dev *dev;
+	struct upf_gtp_dev *dev_any = NULL;
+	struct osmo_sockaddr needle = *local_addr;
+
+	llist_for_each_entry(dev, &g_upf->tunend.devs, entry) {
+		/* To leave the port number out of the cmp, set the needle's port to match */
+		osmo_sockaddr_set_port(&needle.u.sa, osmo_sockaddr_port(&dev->gtpv1.local_addr.u.sa));
+
+		if (!osmo_sockaddr_cmp(&needle, &dev->gtpv1.local_addr))
+			return dev;
+		if (osmo_sockaddr_is_any(&dev->gtpv1.local_addr) == 1)
+			dev_any = dev;
+	}
+	/* No 1:1 match found, but there is a dev listening on ANY? Return that.
+	 * If there is no such dev, return NULL. */
+	return dev_any;
+}
+
 struct upf_gtp_dev *upf_gtp_dev_first()
 {
-	return llist_first_entry_or_null(&g_upf->gtp.devs, struct upf_gtp_dev, entry);
+	return llist_first_entry_or_null(&g_upf->tunend.devs, struct upf_gtp_dev, entry);
 }
 
 /* Tell the kernel to remove the GTP device. Called implicitly by talloc_free() (see upf_gtp_dev_destruct()). */
@@ -96,7 +116,7 @@ static int upf_gtp_dev_delete(struct upf_gtp_dev *dev)
 
 static int upf_gtp_dev_destruct(struct upf_gtp_dev *dev);
 
-/* Allocate state for one GTP device, add to g_upf->gtp.devs and return the created device. If state for the device of
+/* Allocate state for one GTP device, add to g_upf->tunend.devs and return the created device. If state for the device of
  * that name already exists, do nothing and return NULL. */
 static struct upf_gtp_dev *upf_gtp_dev_alloc(const char *name, const char *local_addr)
 {
@@ -124,7 +144,7 @@ static struct upf_gtp_dev *upf_gtp_dev_alloc(const char *name, const char *local
 
 	/* Need to add to list before setting up the destructor. A talloc_free() does automagically remove from the
 	 * list. */
-	llist_add(&dev->entry, &g_upf->gtp.devs);
+	llist_add(&dev->entry, &g_upf->tunend.devs);
 
 	talloc_set_destructor(dev, upf_gtp_dev_destruct);
 
@@ -143,7 +163,7 @@ static int dev_resolve_ifidx(struct upf_gtp_dev *dev)
 	}
 	/* Let's try something to see if talking to the device works. */
 	errno = 0;
-	rc = gtp_list_tunnel(g_upf->gtp.genl_id, g_upf->gtp.nl);
+	rc = gtp_list_tunnel(g_upf->tunend.genl_id, g_upf->tunend.nl);
 	if (errno)
 		rc = -errno;
 	else if (rc)
@@ -172,8 +192,8 @@ int upf_gtp_dev_open(const char *name, bool create_gtp_dev, const char *local_ad
 	int rc;
 	struct upf_gtp_dev *dev;
 
-	if (g_upf->gtp.mockup) {
-		LOGP(DGTP, LOGL_NOTICE, "gtp/mockup active: not opening GTP device '%s'\n", name);
+	if (g_upf->tunend.mockup) {
+		LOGP(DGTP, LOGL_NOTICE, "tunend/mockup active: not opening GTP device '%s'\n", name);
 		return 0;
 	}
 
@@ -183,6 +203,12 @@ int upf_gtp_dev_open(const char *name, bool create_gtp_dev, const char *local_ad
 
 	dev->sgsn_mode = sgsn_mode;
 
+	rc = upf_gtp_genl_ensure_open();
+	if (rc) {
+		LOG_GTP_DEV(dev, LOGL_ERROR, "Cannot set up GTP device, failed to open mnl_socket\n");
+		return rc;
+	}
+
 	if (listen_for_gtpv0) {
 		rc = osmo_sock_init_osa_ofd(&dev->gtpv0.ofd, SOCK_DGRAM, 0, &dev->gtpv0.local_addr, &any,
 					    OSMO_SOCK_F_BIND);
@@ -239,45 +265,40 @@ int upf_gtp_dev_open(const char *name, bool create_gtp_dev, const char *local_ad
 void upf_gtp_devs_close()
 {
 	struct upf_gtp_dev *dev;
-	while ((dev = llist_first_entry_or_null(&g_upf->gtp.devs, struct upf_gtp_dev, entry)))
+	while ((dev = llist_first_entry_or_null(&g_upf->tunend.devs, struct upf_gtp_dev, entry)))
 		talloc_free(dev);
 }
 
 void upf_gtp_genl_close()
 {
-	if (!g_upf->gtp.nl)
+	if (!g_upf->tunend.nl)
 		return;
-	genl_socket_close(g_upf->gtp.nl);
-	g_upf->gtp.nl = NULL;
-	g_upf->gtp.genl_id = -1;
+	genl_socket_close(g_upf->tunend.nl);
+	g_upf->tunend.nl = NULL;
+	g_upf->tunend.genl_id = -1;
 
 	LOGP(DGTP, LOGL_NOTICE, "Closed mnl_socket\n");
 }
 
 /* Open an MNL socket which allows to create and remove GTP devices (requires CAP_NET_ADMIN). */
-int upf_gtp_genl_open()
+int upf_gtp_genl_ensure_open()
 {
-	if (g_upf->gtp.mockup) {
-		LOGP(DGTP, LOGL_NOTICE, "gtp/mockup active: not opening mnl_socket\n");
-		return 0;
-	}
-
 	/* Already open? */
-	if (g_upf->gtp.nl && g_upf->gtp.genl_id >= 0)
+	if (g_upf->tunend.nl && g_upf->tunend.genl_id >= 0)
 		return 0;
 
 	/* sanity / paranoia: if re-opening, make sure the previous socket is closed */
-	if (g_upf->gtp.nl)
+	if (g_upf->tunend.nl)
 		upf_gtp_genl_close();
 
-	g_upf->gtp.nl = genl_socket_open();
-	if (!g_upf->gtp.nl) {
+	g_upf->tunend.nl = genl_socket_open();
+	if (!g_upf->tunend.nl) {
 		LOGP(DGTP, LOGL_ERROR, "Cannot open mnl_socket: %s\n", strerror(errno));
 		return -EIO;
 	}
 
-	g_upf->gtp.genl_id = genl_lookup_family(g_upf->gtp.nl, "gtp");
-	if (g_upf->gtp.genl_id < 0) {
+	g_upf->tunend.genl_id = genl_lookup_family(g_upf->tunend.nl, "gtp");
+	if (g_upf->tunend.genl_id < 0) {
 		LOGP(DGTP, LOGL_ERROR, "genl family 'gtp' not found\n");
 		return -ENOTSUP;
 	}
@@ -286,59 +307,69 @@ int upf_gtp_genl_open()
 	return 0;
 }
 
-struct upf_gtp_tun {
+struct upf_gtp_tunend {
 	struct llist_head entry;
 
 	struct upf_gtp_dev *dev;
-	struct upf_gtp_tun_desc desc;
+	struct upf_tunend desc;
 	bool active;
 };
 
-static int upf_gtp_tun_to_str_buf(char *buf, size_t buflen, const struct upf_gtp_tun *tun)
+static int upf_gtp_tunend_to_str_buf(char *buf, size_t buflen, const struct upf_gtp_tunend *tun)
 {
 	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
-	OSMO_STRBUF_PRINTF(sb, "%s:tun{TEID=l:0x%x,r:0x%x UE=", tun->dev->name, tun->desc.local_teid,
-			   tun->desc.remote_teid);
-	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tun->desc.ue_addr);
-	OSMO_STRBUF_PRINTF(sb, " GTP-dst=");
-	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tun->desc.gtp_remote_addr);
-	OSMO_STRBUF_PRINTF(sb, "}");
+	/* "tunend{dev=apn0 access(GTP-r=1.2.3.4 TEID:l=0x1234,r=0x5678) core(UE-l=10.9.8.7)}" */
+	OSMO_STRBUF_PRINTF(sb, "tunend{dev=%s access(GTP-r=", tun->dev->name);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tun->desc.access.remote.addr);
+	OSMO_STRBUF_PRINTF(sb, " TEID:l=0x%x,r=0x%x) core(UE-l=",
+			   tun->desc.access.local.teid, tun->desc.access.remote.teid);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tun->desc.core.ue_local_addr);
+	OSMO_STRBUF_PRINTF(sb, ")}");
 	return sb.chars_needed;
 }
 
-static char *upf_gtp_tun_to_str_c(void *ctx, const struct upf_gtp_tun *tun)
+static char *upf_gtp_tunend_to_str_c(void *ctx, const struct upf_gtp_tunend *tun)
 {
-	OSMO_NAME_C_IMPL(ctx, 64, "ERROR", upf_gtp_tun_to_str_buf, tun)
+	OSMO_NAME_C_IMPL(ctx, 64, "ERROR", upf_gtp_tunend_to_str_buf, tun)
 }
 
-static int upf_gtp_tun_deactivate(struct upf_gtp_tun *tun);
+static int upf_gtp_tunend_deactivate(struct upf_gtp_tunend *tun);
 
-static int upf_gtp_tun_destruct(struct upf_gtp_tun *tun)
+static int upf_gtp_tunend_destruct(struct upf_gtp_tunend *tun)
 {
 	if (tun->active)
-		upf_gtp_tun_deactivate(tun);
+		upf_gtp_tunend_deactivate(tun);
 	llist_del(&tun->entry);
 	return 0;
 }
 
-static struct upf_gtp_tun *upf_gtp_tun_alloc(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *desc)
+#define tunend_validate(TUNEND) \
+	do { \
+		OSMO_ASSERT(osmo_sockaddr_port(&(TUNEND)->access.local.addr.u.sa) == 0); \
+		OSMO_ASSERT(osmo_sockaddr_port(&(TUNEND)->access.remote.addr.u.sa) == 0); \
+		OSMO_ASSERT(osmo_sockaddr_port(&(TUNEND)->core.ue_local_addr.u.sa) == 0); \
+	} while (0)
+
+static struct upf_gtp_tunend *upf_gtp_tunend_alloc(struct upf_gtp_dev *dev, const struct upf_tunend *desc)
 {
-	struct upf_gtp_tun *tun = talloc(dev, struct upf_gtp_tun);
+	struct upf_gtp_tunend *tun = talloc(dev, struct upf_gtp_tunend);
 	OSMO_ASSERT(tun);
-	*tun = (struct upf_gtp_tun){
+	tunend_validate(desc);
+	*tun = (struct upf_gtp_tunend){
 		.dev = dev,
 		.desc = *desc,
 	};
 	llist_add(&tun->entry, &dev->tunnels);
-	talloc_set_destructor(tun, upf_gtp_tun_destruct);
+	talloc_set_destructor(tun, upf_gtp_tunend_destruct);
 	return tun;
 }
 
-static struct gtp_tunnel *upf_gtp_tun_to_gtp_tunnel(struct upf_gtp_tun *tun)
+static struct gtp_tunnel *upf_gtp_tunend_to_gtp_tunnel(struct upf_gtp_tunend *tun)
 {
 	struct gtp_tunnel *t;
 
-	if (tun->desc.ue_addr.u.sas.ss_family != AF_INET || tun->desc.gtp_remote_addr.u.sas.ss_family != AF_INET) {
+	if (tun->desc.core.ue_local_addr.u.sas.ss_family != AF_INET
+	    || tun->desc.access.remote.addr.u.sas.ss_family != AF_INET) {
 		LOG_GTP_TUN(tun, LOGL_ERROR, "Only capabale of IPv4\n");
 		return NULL;
 	}
@@ -347,14 +378,14 @@ static struct gtp_tunnel *upf_gtp_tun_to_gtp_tunnel(struct upf_gtp_tun *tun)
 	OSMO_ASSERT(t);
 	gtp_tunnel_set_ifidx(t, tun->dev->ifidx);
 	gtp_tunnel_set_version(t, GTP_V1);
-	gtp_tunnel_set_i_tei(t, tun->desc.local_teid);
-	gtp_tunnel_set_o_tei(t, tun->desc.remote_teid);
-	gtp_tunnel_set_ms_ip4(t, &tun->desc.ue_addr.u.sin.sin_addr);
-	gtp_tunnel_set_sgsn_ip4(t, &tun->desc.gtp_remote_addr.u.sin.sin_addr);
+	gtp_tunnel_set_i_tei(t, tun->desc.access.local.teid);
+	gtp_tunnel_set_o_tei(t, tun->desc.access.remote.teid);
+	gtp_tunnel_set_sgsn_ip4(t, &tun->desc.access.remote.addr.u.sin.sin_addr);
+	gtp_tunnel_set_ms_ip4(t, &tun->desc.core.ue_local_addr.u.sin.sin_addr);
 	return t;
 }
 
-int upf_gtp_tun_activate(struct upf_gtp_tun *tun)
+int upf_gtp_tunend_activate(struct upf_gtp_tunend *tun)
 {
 	int rc;
 	struct gtp_tunnel *t;
@@ -362,12 +393,12 @@ int upf_gtp_tun_activate(struct upf_gtp_tun *tun)
 	if (tun->active)
 		return -EALREADY;
 
-	t = upf_gtp_tun_to_gtp_tunnel(tun);
+	t = upf_gtp_tunend_to_gtp_tunnel(tun);
 	if (!t)
 		return -ENOTSUP;
 
 	errno = 0;
-	rc = gtp_add_tunnel(g_upf->gtp.genl_id, g_upf->gtp.nl, t);
+	rc = gtp_add_tunnel(g_upf->tunend.genl_id, g_upf->tunend.nl, t);
 	if (errno) {
 		rc = -errno;
 	} else if (rc) {
@@ -380,37 +411,40 @@ int upf_gtp_tun_activate(struct upf_gtp_tun *tun)
 	return rc;
 }
 
-static struct upf_gtp_tun *upf_gtp_dev_tunnel_find(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *tun_desc)
+static struct upf_gtp_tunend *upf_gtp_dev_tunend_find(struct upf_gtp_dev *dev, const struct upf_tunend *tunend)
 {
-	struct upf_gtp_tun *tun;
+	struct upf_gtp_tunend *tun;
+	tunend_validate(tunend);
 	llist_for_each_entry(tun, &dev->tunnels, entry) {
-		if (upf_gtp_tun_desc_cmp(tun_desc, &tun->desc))
+		if (upf_gtp_tunend_cmp(tunend, &tun->desc))
 			continue;
 		return tun;
 	}
 	return NULL;
 }
 
-int upf_gtp_dev_tunnel_add(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *tun_desc)
+int upf_gtp_dev_tunend_add(struct upf_gtp_dev *dev, const struct upf_tunend *tunend)
 {
-	struct upf_gtp_tun *tun;
-	tun = upf_gtp_dev_tunnel_find(dev, tun_desc);
+	struct upf_gtp_tunend *tun;
+	tunend_validate(tunend);
+	tun = upf_gtp_dev_tunend_find(dev, tunend);
 	if (!tun)
-		tun = upf_gtp_tun_alloc(dev, tun_desc);
+		tun = upf_gtp_tunend_alloc(dev, tunend);
 	if (tun->active)
 		return 0;
-	return upf_gtp_tun_activate(tun);
+	return upf_gtp_tunend_activate(tun);
 }
 
-int upf_gtp_dev_tunnel_del(struct upf_gtp_dev *dev, const struct upf_gtp_tun_desc *tun_desc)
+int upf_gtp_dev_tunend_del(struct upf_gtp_dev *dev, const struct upf_tunend *tunend)
 {
-	struct upf_gtp_tun *tun;
+	struct upf_gtp_tunend *tun;
 	int rc;
-	tun = upf_gtp_dev_tunnel_find(dev, tun_desc);
+	tunend_validate(tunend);
+	tun = upf_gtp_dev_tunend_find(dev, tunend);
 	if (!tun)
 		return 0;
 	if (tun->active) {
-		rc = upf_gtp_tun_deactivate(tun);
+		rc = upf_gtp_tunend_deactivate(tun);
 		if (rc)
 			return rc;
 	}
@@ -418,7 +452,7 @@ int upf_gtp_dev_tunnel_del(struct upf_gtp_dev *dev, const struct upf_gtp_tun_des
 	return 0;
 }
 
-static int upf_gtp_tun_deactivate(struct upf_gtp_tun *tun)
+static int upf_gtp_tunend_deactivate(struct upf_gtp_tunend *tun)
 {
 	int rc;
 	struct gtp_tunnel *t;
@@ -428,13 +462,13 @@ static int upf_gtp_tun_deactivate(struct upf_gtp_tun *tun)
 		return -EINVAL;
 	}
 
-	t = upf_gtp_tun_to_gtp_tunnel(tun);
+	t = upf_gtp_tunend_to_gtp_tunnel(tun);
 	if (!t)
 		return -EINVAL;
 
-	rc = gtp_del_tunnel(g_upf->gtp.genl_id, g_upf->gtp.nl, t);
+	rc = gtp_del_tunnel(g_upf->tunend.genl_id, g_upf->tunend.nl, t);
 	if (rc)
-		LOG_GTP_TUN(tun, LOGL_ERROR, "Failed to delete tunnel: %d %s\n", rc, strerror(rc));
+		LOG_GTP_TUN(tun, LOGL_ERROR, "Failed to delete tunnel\n");
 	else
 		tun->active = false;
 
@@ -444,9 +478,9 @@ static int upf_gtp_tun_deactivate(struct upf_gtp_tun *tun)
 
 static int upf_gtp_dev_destruct(struct upf_gtp_dev *dev)
 {
-	struct upf_gtp_tun *t;
+	struct upf_gtp_tunend *t;
 	/* Destruct and clean up all active tunnels before deleting the device */
-	while ((t = llist_first_entry_or_null(&dev->tunnels, struct upf_gtp_tun, entry)))
+	while ((t = llist_first_entry_or_null(&dev->tunnels, struct upf_gtp_tunend, entry)))
 		talloc_free(t);
 	llist_del(&dev->entry);
 	/* osmo_fd_close() is a noop if ofd.fd == -1 */
@@ -457,7 +491,7 @@ static int upf_gtp_dev_destruct(struct upf_gtp_dev *dev)
 	return 0;
 }
 
-int upf_gtp_tun_desc_cmp(const struct upf_gtp_tun_desc *a, const struct upf_gtp_tun_desc *b)
+int upf_gtp_tunend_cmp(const struct upf_tunend *a, const struct upf_tunend *b)
 {
 	int r;
 
@@ -469,9 +503,9 @@ int upf_gtp_tun_desc_cmp(const struct upf_gtp_tun_desc *a, const struct upf_gtp_
 		return 1;
 
 #define CMP_MEMB(MEMB) OSMO_CMP(a->MEMB, b->MEMB)
-	if ((r = CMP_MEMB(local_teid)))
+	if ((r = CMP_MEMB(access.local.teid)))
 		return r;
-	if ((r = CMP_MEMB(remote_teid)))
+	if ((r = CMP_MEMB(access.remote.teid)))
 		return r;
-	return osmo_sockaddr_cmp(&a->gtp_remote_addr, &b->gtp_remote_addr);
+	return osmo_sockaddr_cmp(&a->access.remote.addr, &b->access.remote.addr);
 }

src/osmo-upf/upf_gtpu_echo.c

@@ -31,6 +31,7 @@ struct gtp1u_hdr {
 		pt:1, /*< Protocol Type: GTP=1, GTP'=0 */
 		version:3; /*< Version: 1 */
 #elif OSMO_IS_BIG_ENDIAN
+/* auto-generated from the little endian part above (libosmocore/contrib/struct_endianness.py) */
 	uint8_t version:3, pt:1, spare:1, e:1, s:1, pn:1;
 #endif
 	uint8_t msg_type;
@@ -83,7 +84,7 @@ static int tx_echo_resp(struct upf_gtp_dev *dev, const struct osmo_sockaddr *rem
 
 	/* ECHO RESPONSE shall contain a recovery counter */
 	msgb_put_u8(msg, GTP1U_IEI_RECOVERY);
-	msgb_put_u8(msg, g_upf->gtp.recovery_count);
+	msgb_put_u8(msg, g_upf->tunend.recovery_count);
 
 	osmo_store16be(msg->tail - tx_h->data1, &tx_h->length);
 
@@ -155,7 +156,12 @@ int upf_gtpu_echo_setup(struct upf_gtp_dev *dev)
 		return -EINVAL;
 	}
 
+	/* the caller should already have osmo_fd_register()ed when setting up the socket. */
+	OSMO_ASSERT(osmo_fd_is_registered(&dev->gtpv1.ofd));
+	/* make sure there is no cb yet that this would be replacing. */
+	OSMO_ASSERT(dev->gtpv1.ofd.cb == NULL);
+
 	dev->gtpv1.ofd.cb = upf_gtpu_echo_read_cb;
 	dev->gtpv1.ofd.data = dev;
-	return osmo_fd_register(&dev->gtpv1.ofd);
+	return 0;
 }

src/osmo-upf/upf_nft.c

@@ -26,122 +26,351 @@
 
 #include <osmocom/core/talloc.h>
 #include <osmocom/core/logging.h>
+#include <osmocom/core/timer.h>
 
 #include <osmocom/upf/upf.h>
 #include <osmocom/upf/upf_nft.h>
 
 static char *upf_nft_ruleset_table_create(void *ctx, const char *table_name)
 {
-	return talloc_asprintf(ctx, "add table inet %s\n", table_name);
+	return talloc_asprintf(ctx, "add table inet %s { flags owner; };\n", table_name);
 }
 
-static int upf_nft_run(const char *ruleset)
+static char *upf_nft_ruleset_vmap_init(void *ctx, const char *table_name, int priority_pre, int priority_post)
+{
+	/* add chain inet osmo-upf pre { type filter hook prerouting priority -300; policy accept; }
+	 * add chain inet osmo-upf post { type filter hook postrouting priority 400; policy accept; }
+	 * add map inet osmo-upf tunmap-pre { typeof ip daddr . @ih,32,32 : verdict; }
+	 * add map inet osmo-upf tunmap-post { typeof meta mark : verdict; }
+	 * add rule inet osmo-upf pre udp dport 2152 ip daddr . @ih,32,32 vmap @tunmap-pre
+	 * add rule inet osmo-upf post meta mark vmap @tunmap-post
+	 */
+	return talloc_asprintf(ctx,
+		"add chain inet %s pre { type filter hook prerouting priority %d; policy accept; };\n"
+		"add chain inet %s post { type filter hook postrouting priority %d; policy accept; };\n"
+		"add map inet %s tunmap-pre { typeof ip daddr . @ih,32,32 : verdict; };\n"
+		"add map inet %s tunmap-post { typeof meta mark : verdict; };\n"
+		"add rule inet %s pre udp dport %u ip daddr . @ih,32,32 vmap @tunmap-pre;\n"
+		"add rule inet %s post meta mark vmap @tunmap-post;\n",
+		table_name, priority_pre,
+		table_name, priority_post,
+		table_name,
+		table_name,
+		table_name, PORT_GTP1_U,
+		table_name);
+}
+
+static int upf_nft_run_now(const char *ruleset)
 {
 	int rc;
+	const int logmax = 256;
 
-	if (g_upf->nft.mockup) {
-		LOGP(DNFT, LOGL_NOTICE, "nft/mockup active: not running nft ruleset: '%s'\n", ruleset);
+	if (g_upf->tunmap.mockup) {
+		LOGP(DNFT, LOGL_NOTICE, "tunmap/mockup active: not running nft ruleset: '%s'\n", ruleset);
 		return 0;
 	}
 
-	if (!g_upf->nft.nft_ctx) {
+	if (!g_upf->tunmap.nft_ctx) {
 		rc = upf_nft_init();
 		if (rc)
 			return rc;
 	}
 
-	rc = nft_run_cmd_from_buffer(g_upf->nft.nft_ctx, ruleset);
+	rc = nft_run_cmd_from_buffer(g_upf->tunmap.nft_ctx, ruleset);
 	if (rc < 0) {
 		LOGP(DNFT, LOGL_ERROR, "error running nft ruleset: rc=%d ruleset=%s\n",
 		     rc, osmo_quote_str_c(OTC_SELECT, ruleset, -1));
 		return -EIO;
 	}
+
+	if (log_check_level(DNFT, LOGL_DEBUG)) {
+		size_t l = strlen(ruleset);
+		LOGP(DNFT, LOGL_DEBUG, "ran nft ruleset, %zu chars: \"%s%s\"\n",
+		     l,
+		     osmo_escape_cstr_c(OTC_SELECT, ruleset, OSMO_MIN(logmax, l)),
+		     l > logmax ? "..." : "");
+	}
+
 	return 0;
 }
 
+struct nft_queue {
+	struct osmo_tdef *flush_time_tdef;
+	struct osmo_tdef *ruleset_max_tdef;
+	struct osmo_strbuf sb;
+	/* 128 NFT rulesets amount to about 110 kb of char */
+	char buf[1<<17];
+	unsigned int ruleset_count;
+	struct osmo_timer_list timer;
+};
+
+static void nft_queue_clear_buf(struct nft_queue *q)
+{
+	q->sb = (struct osmo_strbuf){ .buf = q->buf, .len = sizeof(q->buf) };
+	q->buf[0] = '\0';
+}
+
+static void nft_queue_init(void *ctx, struct nft_queue *q,
+			   struct osmo_tdef *flush_time_tdef,
+			   struct osmo_tdef *ruleset_max_tdef)
+{
+	*q = (struct nft_queue){
+		.flush_time_tdef = flush_time_tdef,
+		.ruleset_max_tdef = ruleset_max_tdef,
+	};
+	nft_queue_clear_buf(q);
+}
+
+static void nft_queue_flush(struct nft_queue *q, const char *reason)
+{
+	static unsigned int flush_count = 0;
+	static unsigned int ruleset_count = 0;
+
+	/* We will now flush the queue empty. A timer needs to run only when the next pending entry is added. */
+	osmo_timer_del(&q->timer);
+
+	/* Nothing to send? */
+	if (!q->sb.chars_needed)
+		return;
+
+	flush_count++;
+	ruleset_count += q->ruleset_count;
+	LOGP(DNFT, LOGL_INFO, "Flushing NFT ruleset queue: %s: n:%u strlen:%zu (flush count: %u avg rules per flush: %s)\n",
+	     reason,
+	     q->ruleset_count, q->sb.chars_needed,
+	     flush_count, osmo_int_to_float_str_c(OTC_SELECT, 10 * ruleset_count / flush_count, 1));
+
+	q->ruleset_count = 0;
+
+	upf_nft_run_now(q->sb.buf);
+
+	nft_queue_clear_buf(q);
+}
+
+static void nft_queue_flush_cb(void *q)
+{
+	nft_queue_flush(q, "timeout");
+}
+
+static int nft_enqueue(struct nft_queue *q,
+		       int (*tunmap_to_str_buf)(char *buf, size_t len, struct upf_tunmap *tunmap),
+		       struct upf_tunmap *tunmap)
+{
+	int ruleset_max;
+	struct osmo_strbuf q_sb_was = q->sb;
+
+	OSMO_STRBUF_APPEND(q->sb, tunmap_to_str_buf, tunmap);
+
+	/* is that being cut off? then revert the addition. This should never happen in practice. */
+	if (q->sb.chars_needed >= q->sb.len) {
+		q->sb = q_sb_was;
+		if (q->sb.pos)
+			*q->sb.pos = '\0';
+		nft_queue_flush(q, "reached max nr of chars");
+		OSMO_STRBUF_APPEND(q->sb, tunmap_to_str_buf, tunmap);
+	}
+
+	/* Append separator -- no problem if that gets cut off. */
+	OSMO_STRBUF_PRINTF(q->sb, "\n");
+
+	q->ruleset_count++;
+
+	LOGP(DNFT, LOGL_INFO, "Added NFT ruleset to queue: n:%u strlen:%zu\n",
+	     q->ruleset_count, q->sb.chars_needed);
+
+	/* Added a rule, see if it has reached ruleset_max. */
+	ruleset_max = osmo_tdef_get(q->ruleset_max_tdef, q->ruleset_max_tdef->T, OSMO_TDEF_CUSTOM, 128);
+	if (q->ruleset_count >= ruleset_max) {
+		nft_queue_flush(q, "reached max nr of rules");
+		return 0;
+	}
+
+	/* Item added. If the timer is not running yet, schedule a flush in given timeout */
+	if (!osmo_timer_pending(&q->timer)) {
+		struct osmo_tdef *t;
+		unsigned long us;
+		osmo_timer_setup(&q->timer, nft_queue_flush_cb, q);
+		t = q->flush_time_tdef;
+		us = osmo_tdef_get(t, t->T, OSMO_TDEF_US, 100000);
+		osmo_timer_schedule(&q->timer, us / 1000000, us % 1000000);
+	}
+	return 0;
+}
+
+static void nft_queue_free(struct nft_queue *q)
+{
+	osmo_timer_del(&q->timer);
+}
+
+static struct nft_queue g_nft_queue = {};
+
 int upf_nft_init()
 {
 	int rc;
-	if (g_upf->nft.mockup) {
+
+	nft_queue_init(g_upf, &g_nft_queue,
+		       osmo_tdef_get_entry(g_upf_nft_tdefs, -32),
+		       osmo_tdef_get_entry(g_upf_nft_tdefs, -33));
+
+	/* Always set up the default settings, also in mockup mode, so that the VTY reflects sane values */
+	if (!g_upf->tunmap.table_name)
+		g_upf->tunmap.table_name = talloc_strdup(g_upf, "osmo-upf");
+
+	/* When in mockup mode, do not set up nft_ctx and netfilter table */
+	if (g_upf->tunmap.mockup) {
 		LOGP(DNFT, LOGL_NOTICE,
-		     "nft/mockup active: not allocating libnftables nft_ctx. FOR TESTING PURPOSES ONLY.\n");
+		     "tunmap/mockup active: not allocating libnftables nft_ctx. FOR TESTING PURPOSES ONLY.\n");
 		return 0;
 	}
 
-	g_upf->nft.nft_ctx = nft_ctx_new(NFT_CTX_DEFAULT);
-	if (!g_upf->nft.nft_ctx) {
+	g_upf->tunmap.nft_ctx = nft_ctx_new(NFT_CTX_DEFAULT);
+	if (!g_upf->tunmap.nft_ctx) {
 		LOGP(DNFT, LOGL_ERROR, "cannot allocate libnftables nft_ctx\n");
 		return -EIO;
 	}
 
-	if (!g_upf->nft.table_name)
-		g_upf->nft.table_name = talloc_strdup(g_upf, "osmo-upf");
-
-	rc = upf_nft_run(upf_nft_ruleset_table_create(OTC_SELECT, g_upf->nft.table_name));
+	rc = upf_nft_run_now(upf_nft_tunmap_get_table_init_str(OTC_SELECT));
 	if (rc) {
 		LOGP(DNFT, LOGL_ERROR, "Failed to create nft table %s\n",
-		     osmo_quote_str_c(OTC_SELECT, g_upf->nft.table_name, -1));
+		     osmo_quote_str_c(OTC_SELECT, g_upf->tunmap.table_name, -1));
+		return rc;
+	}
+	LOGP(DNFT, LOGL_NOTICE, "Created nft table %s\n", osmo_quote_str_c(OTC_SELECT, g_upf->tunmap.table_name, -1));
+
+	rc = upf_nft_run_now(upf_nft_tunmap_get_vmap_init_str(OTC_SELECT));
+	if (rc) {
+		LOGP(DNFT, LOGL_ERROR, "Failed to initialize nft verdict map in table %s\n", g_upf->tunmap.table_name);
 		return rc;
 	}
-	LOGP(DNFT, LOGL_NOTICE, "Created nft table %s\n", osmo_quote_str_c(OTC_SELECT, g_upf->nft.table_name, -1));
 	return 0;
 }
 
 int upf_nft_free()
 {
-	if (!g_upf->nft.nft_ctx)
+	nft_queue_free(&g_nft_queue);
+	if (!g_upf->tunmap.nft_ctx)
 		return 0;
-	nft_ctx_free(g_upf->nft.nft_ctx);
-	g_upf->nft.nft_ctx = NULL;
+	nft_ctx_free(g_upf->tunmap.nft_ctx);
+	g_upf->tunmap.nft_ctx = NULL;
 	return 0;
 }
 
 struct upf_nft_args_peer {
 	/* The source IP address in packets received from this peer */
-	const struct osmo_sockaddr *addr;
+	const struct osmo_sockaddr *addr_remote;
 	/* The TEID that we send to the peer in GTP packets. */
 	uint32_t teid_remote;
+	/* The local destination IP address in packets received from this peer */
+	const struct osmo_sockaddr *addr_local;
 	/* The TEID that the peer sends to us in GTP packets. */
 	uint32_t teid_local;
+	/* The nft chain id that forwards packets received on addr_local,teid_local. Also used for the 'mark' id in
+	 * the verdict map ruleset. */
+	uint32_t chain_id;
 };
 
 struct upf_nft_args {
 	/* global table name */
 	const char *table_name;
-	/* chain name for this specific tunnel mapping */
-	uint32_t chain_id;
-	int priority;
 
 	struct upf_nft_args_peer peer_a;
 	struct upf_nft_args_peer peer_b;
 };
 
-static int tunmap_single_direction(char *buf, size_t buflen,
-				   const struct upf_nft_args *args,
-				   const struct upf_nft_args_peer *from_peer,
-				   const struct upf_nft_args_peer *to_peer)
+static int tunmap_add_single_direction(char *buf, size_t buflen,
+				       const struct upf_nft_args *args,
+				       bool dir_a2b)
 {
 	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
-	OSMO_STRBUF_PRINTF(sb, "add rule inet %s " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u", args->table_name, args->chain_id);
+	const struct upf_nft_args_peer *from_peer;
+	const struct upf_nft_args_peer *to_peer;
 
-	/* Match only UDP packets */
-	OSMO_STRBUF_PRINTF(sb, " meta l4proto udp");
+	if (dir_a2b) {
+		from_peer = &args->peer_a;
+		to_peer = &args->peer_b;
+	} else {
+		from_peer = &args->peer_b;
+		to_peer = &args->peer_a;
+	}
 
-	/* Match on packets coming in from from_peer */
-	OSMO_STRBUF_PRINTF(sb, " ip saddr ");
-	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, from_peer->addr);
+	/* # add chain for verdict map in prerouting
+	 * add chain inet osmo-upf tunmap-pre-123
+	 * # mangle destination address at prerouting
+	 * add rule inet osmo-upf tunmap-pre-123 ip daddr set 1.1.1.1 meta mark set 123 counter accept
+	 *
+	 * # add chain for verdict map in postrouting
+	 * add chain inet osmo-upf tunmap-post-123
+	 * # mangle source address and GTP TID at postrouting
+	 * add rule inet osmo-upf tunmap-post-123 ip saddr set 2.2.2.1 udp sport set 2152 @ih,32,32 set 0x00000102 counter accept
+	 *
+	 * # add elements to verdict map, jump to chain
+	 * add element inet osmo-upf tunmap-pre { 2.2.2.3 . 0x00000203 : jump tunmap-pre-123 }
+	 * add element inet osmo-upf tunmap-post { 123 : jump tunmap-post-123 }
+	 */
 
-	/* Match on the TEID in the header */
-	OSMO_STRBUF_PRINTF(sb, " @ih,32,32 0x%08x", from_peer->teid_local);
+	OSMO_STRBUF_PRINTF(sb, "add chain inet %s tunmap-pre-%u;\n",
+			   args->table_name, from_peer->chain_id);
 
-	/* Change destination address to to_peer */
+	OSMO_STRBUF_PRINTF(sb, "add rule inet %s tunmap-pre-%u",
+			   args->table_name, from_peer->chain_id);
 	OSMO_STRBUF_PRINTF(sb, " ip daddr set ");
-	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, to_peer->addr);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, to_peer->addr_remote);
+	OSMO_STRBUF_PRINTF(sb, " meta mark set %u counter accept;\n", from_peer->chain_id);
 
-	/* Change the TEID in the header to the one to_peer expects */
-	OSMO_STRBUF_PRINTF(sb, " @ih,32,32 set 0x%08x", to_peer->teid_remote);
+	OSMO_STRBUF_PRINTF(sb, "add chain inet %s tunmap-post-%u;\n",
+			   args->table_name, from_peer->chain_id);
 
-	OSMO_STRBUF_PRINTF(sb, " counter\n");
+	OSMO_STRBUF_PRINTF(sb, "add rule inet %s tunmap-post-%u",
+			   args->table_name, from_peer->chain_id);
+	OSMO_STRBUF_PRINTF(sb, " ip saddr set ");
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, to_peer->addr_local);
+	OSMO_STRBUF_PRINTF(sb, " udp sport set 2152");
+	OSMO_STRBUF_PRINTF(sb, " @ih,32,32 set 0x%x", to_peer->teid_remote);
+	OSMO_STRBUF_PRINTF(sb, " counter accept;\n");
+
+	OSMO_STRBUF_PRINTF(sb, "add element inet %s tunmap-pre { ",
+			   args->table_name);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, from_peer->addr_local);
+	OSMO_STRBUF_PRINTF(sb, " . 0x%x : jump tunmap-pre-%u };\n",
+			   from_peer->teid_local, from_peer->chain_id);
+
+	OSMO_STRBUF_PRINTF(sb, "add element inet %s tunmap-post { %u : jump tunmap-post-%u };\n",
+			   args->table_name, from_peer->chain_id, from_peer->chain_id);
+
+	return sb.chars_needed;
+}
+
+static int tunmap_del_single_direction(char *buf, size_t buflen,
+				       const struct upf_nft_args *args,
+				       bool dir_a2b)
+{
+	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
+	const struct upf_nft_args_peer *from_peer;
+
+	if (dir_a2b)
+		from_peer = &args->peer_a;
+	else
+		from_peer = &args->peer_b;
+
+	/* delete element inet osmo-upf tunmap-pre { 2.2.2.3 . 0x203 }
+	 * delete element inet osmo-upf tunmap-post { 123 }
+	 * delete chain inet osmo-upf tunmap-pre-123
+	 * delete chain inet osmo-upf tunmap-post-123
+	 */
+
+	OSMO_STRBUF_PRINTF(sb, "delete element inet %s tunmap-pre { ",
+			   args->table_name);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, from_peer->addr_local);
+	OSMO_STRBUF_PRINTF(sb, " . 0x%x };\n", from_peer->teid_local);
+
+	OSMO_STRBUF_PRINTF(sb, "delete element inet %s tunmap-post { %u };\n",
+			   args->table_name, from_peer->chain_id);
+
+	OSMO_STRBUF_PRINTF(sb, "delete chain inet %s tunmap-pre-%u;\n",
+			   args->table_name, from_peer->chain_id);
+
+	OSMO_STRBUF_PRINTF(sb, "delete chain inet %s tunmap-post-%u;\n",
+			   args->table_name, from_peer->chain_id);
 
 	return sb.chars_needed;
 }
@@ -150,70 +379,128 @@ static int upf_nft_ruleset_tunmap_create_buf(char *buf, size_t buflen, const str
 {
 	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
 
-	/* Add a chain for this tunnel mapping */
-	OSMO_STRBUF_PRINTF(sb, "add chain inet %s " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u { type filter hook prerouting priority %d; }\n",
-			   args->table_name, args->chain_id, args->priority);
-
 	/* Forwarding from peer_a to peer_b */
-	OSMO_STRBUF_APPEND(sb, tunmap_single_direction, args, &args->peer_a, &args->peer_b);
+	OSMO_STRBUF_APPEND(sb, tunmap_add_single_direction, args, true);
 	/* And from peer_b to peer_a */
-	OSMO_STRBUF_APPEND(sb, tunmap_single_direction, args, &args->peer_b, &args->peer_a);
+	OSMO_STRBUF_APPEND(sb, tunmap_add_single_direction, args, false);
 
 	return sb.chars_needed;
 }
 
-static char *upf_nft_ruleset_tunmap_create_c(void *ctx, const struct upf_nft_args *args)
-{
-	OSMO_NAME_C_IMPL(ctx, 512, "ERROR", upf_nft_ruleset_tunmap_create_buf, args)
-}
-
 static int upf_nft_ruleset_tunmap_delete_buf(char *buf, size_t buflen, const struct upf_nft_args *args)
 {
 	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
-	OSMO_STRBUF_PRINTF(sb, "delete chain inet %s " NFT_CHAIN_NAME_PREFIX_TUNMAP "%u\n",
-			   args->table_name, args->chain_id);
+
+	/* Forwarding from peer_a to peer_b */
+	OSMO_STRBUF_APPEND(sb, tunmap_del_single_direction, args, true);
+	/* And from peer_b to peer_a */
+	OSMO_STRBUF_APPEND(sb, tunmap_del_single_direction, args, false);
+
 	return sb.chars_needed;
 }
 
-static char *upf_nft_ruleset_tunmap_delete_c(void *ctx, const struct upf_nft_args *args)
+int upf_nft_tunmap_to_str_buf(char *buf, size_t buflen, const struct upf_tunmap *tunmap)
 {
-	OSMO_NAME_C_IMPL(ctx, 64, "ERROR", upf_nft_ruleset_tunmap_delete_buf, args)
+	struct osmo_strbuf sb = { .buf = buf, .len = buflen };
+
+	/* ACCESS 1.1.1.2:0x102 <---> 2.2.2.1:0x201 UPF 2.2.2.3:0x203 <---> 3.3.3.2:0x302 CORE */
+	OSMO_STRBUF_PRINTF(sb, "ACCESS ");
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tunmap->access.tun.remote.addr);
+	OSMO_STRBUF_PRINTF(sb, ":0x%x <---> ", tunmap->access.tun.remote.teid);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tunmap->access.tun.local.addr);
+	OSMO_STRBUF_PRINTF(sb, ":0x%x UPF ", tunmap->access.tun.local.teid);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tunmap->core.tun.local.addr);
+	OSMO_STRBUF_PRINTF(sb, ":0x%x <---> ", tunmap->core.tun.local.teid);
+	OSMO_STRBUF_APPEND(sb, osmo_sockaddr_to_str_buf2, &tunmap->core.tun.remote.addr);
+	OSMO_STRBUF_PRINTF(sb, ":0x%x CORE", tunmap->core.tun.remote.teid);
+	return sb.chars_needed;
 }
 
-static void upf_nft_args_from_tunmap_desc(struct upf_nft_args *args, const struct upf_nft_tunmap_desc *tunmap)
+char *upf_nft_tunmap_to_str_c(void *ctx, const struct upf_tunmap *tunmap)
 {
+	OSMO_NAME_C_IMPL(ctx, 128, "ERROR", upf_nft_tunmap_to_str_buf, tunmap)
+}
+
+static void upf_nft_args_from_tunmap(struct upf_nft_args *args, const struct upf_tunmap *tunmap)
+{
+	OSMO_ASSERT(osmo_sockaddr_port(&tunmap->access.tun.remote.addr.u.sa) == 0);
+	OSMO_ASSERT(osmo_sockaddr_port(&tunmap->access.tun.local.addr.u.sa) == 0);
+	OSMO_ASSERT(osmo_sockaddr_port(&tunmap->core.tun.remote.addr.u.sa) == 0);
+	OSMO_ASSERT(osmo_sockaddr_port(&tunmap->core.tun.local.addr.u.sa) == 0);
+
 	*args = (struct upf_nft_args){
-		.table_name = g_upf->nft.table_name,
-		.chain_id = tunmap->id,
-		.priority = g_upf->nft.priority,
+		.table_name = g_upf->tunmap.table_name,
 		.peer_a = {
-			.addr = &tunmap->access.gtp_remote_addr,
-			.teid_remote = tunmap->access.remote_teid,
-			.teid_local = tunmap->access.local_teid,
+			.addr_remote = &tunmap->access.tun.remote.addr,
+			.teid_remote = tunmap->access.tun.remote.teid,
+			.addr_local = &tunmap->access.tun.local.addr,
+			.teid_local = tunmap->access.tun.local.teid,
+			.chain_id = tunmap->access.chain_id,
 		},
 		.peer_b = {
-			.addr = &tunmap->core.gtp_remote_addr,
-			.teid_remote = tunmap->core.remote_teid,
-			.teid_local = tunmap->core.local_teid,
+			.addr_remote = &tunmap->core.tun.remote.addr,
+			.teid_remote = tunmap->core.tun.remote.teid,
+			.addr_local = &tunmap->core.tun.local.addr,
+			.teid_local = tunmap->core.tun.local.teid,
+			.chain_id = tunmap->core.chain_id,
 		},
 	};
 }
 
-int upf_nft_tunmap_create(struct upf_nft_tunmap_desc *tunmap)
+char *upf_nft_tunmap_get_table_init_str(void *ctx)
 {
-	struct upf_nft_args args;
-
-	/* Give this tunnel mapping a new id, returned to the caller so that the tunnel mapping can be deleted later */
-	g_upf->nft.next_id_state++;
-	tunmap->id = g_upf->nft.next_id_state;
-
-	upf_nft_args_from_tunmap_desc(&args, tunmap);
-	return upf_nft_run(upf_nft_ruleset_tunmap_create_c(OTC_SELECT, &args));
+	return upf_nft_ruleset_table_create(ctx, g_upf->tunmap.table_name);
 }
 
-int upf_nft_tunmap_delete(struct upf_nft_tunmap_desc *tunmap)
+char *upf_nft_tunmap_get_vmap_init_str(void *ctx)
+{
+	return upf_nft_ruleset_vmap_init(ctx, g_upf->tunmap.table_name, g_upf->tunmap.priority_pre,
+					 g_upf->tunmap.priority_post);
+}
+
+int upf_nft_tunmap_get_ruleset_str_buf(char *buf, size_t len, struct upf_tunmap *tunmap)
 {
 	struct upf_nft_args args;
-	upf_nft_args_from_tunmap_desc(&args, tunmap);
-	return upf_nft_run(upf_nft_ruleset_tunmap_delete_c(OTC_SELECT, &args));
+	upf_nft_args_from_tunmap(&args, tunmap);
+	return upf_nft_ruleset_tunmap_create_buf(buf, len, &args);
+}
+
+char *upf_nft_tunmap_get_ruleset_str(void *ctx, struct upf_tunmap *tunmap)
+{
+	OSMO_NAME_C_IMPL(ctx, 1024, "ERROR", upf_nft_tunmap_get_ruleset_str_buf, tunmap)
+}
+
+int upf_nft_tunmap_get_ruleset_del_str_buf(char *buf, size_t len, struct upf_tunmap *tunmap)
+{
+	struct upf_nft_args args;
+	upf_nft_args_from_tunmap(&args, tunmap);
+	return upf_nft_ruleset_tunmap_delete_buf(buf, len, &args);
+}
+
+char *upf_nft_tunmap_get_ruleset_del_str(void *ctx, struct upf_tunmap *tunmap)
+{
+	OSMO_NAME_C_IMPL(ctx, 1024, "ERROR", upf_nft_tunmap_get_ruleset_del_str_buf, tunmap)
+}
+
+static int upf_nft_tunmap_ensure_chain_id(struct upf_nft_tun *tun)
+{
+	if (tun->chain_id)
+		return 0;
+	tun->chain_id = upf_next_chain_id();
+	if (!tun->chain_id)
+		return -ENOSPC;
+	return 0;
+}
+
+int upf_nft_tunmap_create(struct upf_tunmap *tunmap)
+{
+	if (upf_nft_tunmap_ensure_chain_id(&tunmap->access)
+	    || upf_nft_tunmap_ensure_chain_id(&tunmap->core))
+		return -ENOSPC;
+	return nft_enqueue(&g_nft_queue, upf_nft_tunmap_get_ruleset_str_buf, tunmap);
+}
+
+int upf_nft_tunmap_delete(struct upf_tunmap *tunmap)
+{
+	return nft_enqueue(&g_nft_queue, upf_nft_tunmap_get_ruleset_del_str_buf, tunmap);
 }

src/osmo-upf/upf_vty.c

@@ -37,11 +37,13 @@
 #include <osmocom/upf/up_peer.h>
 #include <osmocom/upf/up_session.h>
 #include <osmocom/upf/up_gtp_action.h>
+#include <osmocom/upf/netinst.h>
 
 enum upf_vty_node {
 	PFCP_NODE = _LAST_OSMOVTY_NODE + 1,
-	GTP_NODE,
-	NFT_NODE,
+	TUNEND_NODE,
+	TUNMAP_NODE,
+	NETINST_NODE,
 };
 
 static struct cmd_node cfg_pfcp_node = {
@@ -51,7 +53,7 @@ static struct cmd_node cfg_pfcp_node = {
 };
 
 #define pfcp_vty (g_upf->pfcp.vty_cfg)
-#define gtp_vty (g_upf->gtp.vty_cfg)
+#define tunend_vty (g_upf->tunend.vty_cfg)
 
 DEFUN(cfg_pfcp, cfg_pfcp_cmd,
       "pfcp",
@@ -78,29 +80,32 @@ DEFUN(cfg_pfcp_local_addr, cfg_pfcp_local_addr_cmd,
 	return CMD_SUCCESS;
 }
 
-static struct cmd_node cfg_gtp_node = {
-	GTP_NODE,
-	"%s(config-gtp)# ",
+static struct cmd_node cfg_tunend_node = {
+	TUNEND_NODE,
+	"%s(config-tunend)# ",
 	1,
 };
 
-DEFUN(cfg_gtp, cfg_gtp_cmd,
-      "gtp",
-      "Enter the 'gtp' node to configure Linux GTP kernel module usage\n")
+#define TUNEND_NODE_STR "Enter the 'tunend' node to configure Linux GTP kernel module usage\n"
+
+DEFUN(cfg_tunend, cfg_tunend_cmd, "tunend", TUNEND_NODE_STR)
 {
-	vty->node = GTP_NODE;
+	vty->node = TUNEND_NODE;
 	return CMD_SUCCESS;
 }
 
-static int config_write_gtp(struct vty *vty)
-{
-	struct gtp_vty_cfg_dev *d;
-	vty_out(vty, "gtp%s", VTY_NEWLINE);
+/* legacy compat: "tunend" was originally named "gtp" */
+DEFUN_CMD_ELEMENT(cfg_tunend, cfg_gtp_cmd, "gtp", TUNEND_NODE_STR, CMD_ATTR_HIDDEN, 0);
 
-	if (g_upf->gtp.mockup)
+static int config_write_tunend(struct vty *vty)
+{
+	struct tunend_vty_cfg_dev *d;
+	vty_out(vty, "tunend%s", VTY_NEWLINE);
+
+	if (g_upf->tunend.mockup)
 		vty_out(vty, " mockup%s", VTY_NEWLINE);
 
-	llist_for_each_entry(d, &gtp_vty.devs, entry) {
+	llist_for_each_entry(d, &tunend_vty.devs, entry) {
 		if (d->create) {
 			vty_out(vty, " dev create %s", d->dev_name);
 			if (d->local_addr)
@@ -115,69 +120,76 @@ static int config_write_gtp(struct vty *vty)
 
 #define DEV_STR "Configure the GTP device to use for encaps/decaps.\n"
 
-DEFUN(cfg_gtp_mockup, cfg_gtp_mockup_cmd,
+DEFUN(cfg_tunend_mockup, cfg_tunend_mockup_cmd,
       "mockup",
       "don't actually send commands to the GTP kernel module, just return success\n")
 {
-	g_upf->gtp.mockup = true;
+	g_upf->tunend.mockup = true;
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_gtp_no_mockup, cfg_gtp_no_mockup_cmd,
+DEFUN(cfg_tunend_no_mockup, cfg_tunend_no_mockup_cmd,
       "no mockup",
       NO_STR
       "operate GTP kernel module normally\n")
 {
-	g_upf->gtp.mockup = false;
+	g_upf->tunend.mockup = false;
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_gtp_dev_create, cfg_gtp_dev_create_cmd,
+static struct tunend_vty_cfg_dev *tunend_dev_add(int argc, const char **argv, bool create)
+{
+	struct tunend_vty_cfg_dev *d = talloc_zero(g_upf, struct tunend_vty_cfg_dev);
+	d->create = create;
+	d->dev_name = talloc_strdup(d, argv[0]);
+	if (argc > 1)
+		d->local_addr = talloc_strdup(d, argv[1]);
+	llist_add(&d->entry, &tunend_vty.devs);
+	return d;
+}
+
+DEFUN(cfg_tunend_dev_create, cfg_tunend_dev_create_cmd,
       "dev create DEVNAME [LISTEN_ADDR]",
       DEV_STR
       "Add GTP device, creating a new Linux kernel GTP device. Will listen on GTPv1 port "
       OSMO_STRINGIFY_VAL(PORT_GTP1_U)
-      " and GTPv0 port " OSMO_STRINGIFY_VAL(PORT_GTP0_U) " on the specified interface, or on ANY if LISTEN_ADDR is"
-      " omitted.\n"
+      " and GTPv0 port " OSMO_STRINGIFY_VAL(PORT_GTP0_U) " on the specified LISTEN_ADDR\n"
       "device name, e.g. 'apn0'\n"
-      "IPv4 or IPv6 address to listen on, omit for ANY\n")
+      "IPv4 or IPv6 address to listen on, omit for ANY. LISTEN_ADDR is used to pick a GTP device matching the local"
+      " address for a PFCP Network Instance, which are configured in the 'netinst' node.\n")
 {
-	struct gtp_vty_cfg_dev *d = talloc_zero(g_upf, struct gtp_vty_cfg_dev);
-	d->create = true;
-	d->dev_name = talloc_strdup(d, argv[0]);
-	if (argc > 1)
-		d->local_addr = talloc_strdup(d, argv[1]);
-	llist_add(&d->entry, &gtp_vty.devs);
-	vty_out(vty, "Added GTP device %s (create new)%s", d->dev_name, VTY_NEWLINE);
+	struct tunend_vty_cfg_dev *d = tunend_dev_add(argc, argv, true);
+	vty_out(vty, "Added GTP device %s on %s (create new)%s", d->dev_name, d->local_addr ? : "0.0.0.0", VTY_NEWLINE);
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_gtp_dev_use, cfg_gtp_dev_use_cmd,
-      "dev use DEVNAME",
+DEFUN(cfg_tunend_dev_use, cfg_tunend_dev_use_cmd,
+      "dev use DEVNAME [LOCAL_ADDR]",
       DEV_STR
       "Add GTP device, using an existing Linux kernel GTP device, e.g. created by 'gtp-link'\n"
-      "device name, e.g. 'apn0'\n")
+      "device name, e.g. 'apn0'\n"
+      "The local GTP address this device listens on. It is assumed to be ANY when omitted."
+      " LOCAL_ADDR is used to pick a GTP device matching the local address for a PFCP Network Instance,"
+      " which are configured in the 'netinst' node.\n")
 {
-	struct gtp_vty_cfg_dev *d = talloc_zero(g_upf, struct gtp_vty_cfg_dev);
-	d->create = false;
-	d->dev_name = talloc_strdup(d, argv[0]);
-	llist_add(&d->entry, &gtp_vty.devs);
-	vty_out(vty, "Added GTP device %s (use existing)%s", d->dev_name, VTY_NEWLINE);
+	struct tunend_vty_cfg_dev *d = tunend_dev_add(argc, argv, false);
+	vty_out(vty, "Added GTP device %s on %s (use existing)%s", d->dev_name, d->local_addr ? : "0.0.0.0",
+		VTY_NEWLINE);
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_gtp_dev_del, cfg_gtp_dev_del_cmd,
+DEFUN(cfg_tunend_dev_del, cfg_tunend_dev_del_cmd,
       "dev delete DEVNAME",
       DEV_STR
       "Remove a GTP device from the configuration, and delete the Linux kernel GTP device if it was created here.\n"
       "device name, e.g. 'apn0'\n")
 {
 	const char *dev_name = argv[0];
-	struct gtp_vty_cfg_dev *d;
+	struct tunend_vty_cfg_dev *d;
 	struct upf_gtp_dev *dev;
 
 	/* remove from VTY cfg */
-	llist_for_each_entry(d, &gtp_vty.devs, entry) {
+	llist_for_each_entry(d, &tunend_vty.devs, entry) {
 		if (strcmp(d->dev_name, dev_name))
 			continue;
 		llist_del(&d->entry);
@@ -191,57 +203,197 @@ DEFUN(cfg_gtp_dev_del, cfg_gtp_dev_del_cmd,
 	return CMD_SUCCESS;
 }
 
-static struct cmd_node cfg_nft_node = {
-	NFT_NODE,
-	"%s(config-nft)# ",
+static struct cmd_node cfg_tunmap_node = {
+	TUNMAP_NODE,
+	"%s(config-tunmap)# ",
 	1,
 };
 
-DEFUN(cfg_nft, cfg_nft_cmd,
-      "nft",
-      "Enter the 'nft' node to configure nftables usage\n")
+#define TUNMAP_NODE_STR "Enter the 'tunmap' node to configure nftables usage\n"
+
+DEFUN(cfg_tunmap, cfg_tunmap_cmd, "tunmap", TUNMAP_NODE_STR)
 {
-	vty->node = NFT_NODE;
+	vty->node = TUNMAP_NODE;
 	return CMD_SUCCESS;
 }
 
-static int config_write_nft(struct vty *vty)
-{
-	vty_out(vty, "nft%s", VTY_NEWLINE);
+/* legacy compat: "tunmap" was originally named "nft" */
+DEFUN_CMD_ELEMENT(cfg_tunmap, cfg_nft_cmd, "nft", TUNMAP_NODE_STR, CMD_ATTR_HIDDEN, 0);
 
-	if (g_upf->nft.mockup)
+static int config_write_tunmap(struct vty *vty)
+{
+	vty_out(vty, "tunmap%s", VTY_NEWLINE);
+
+	if (g_upf->tunmap.mockup)
 		vty_out(vty, " mockup%s", VTY_NEWLINE);
 
-	if (g_upf->nft.table_name && strcmp(g_upf->nft.table_name, "osmo-upf"))
-		vty_out(vty, " table-name %s%s", g_upf->nft.table_name, VTY_NEWLINE);
+	if (g_upf->tunmap.table_name && strcmp(g_upf->tunmap.table_name, "osmo-upf"))
+		vty_out(vty, " table-name %s%s", g_upf->tunmap.table_name, VTY_NEWLINE);
+
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_nft_mockup, cfg_nft_mockup_cmd,
+DEFUN(cfg_tunmap_mockup, cfg_tunmap_mockup_cmd,
       "mockup",
       "don't actually send rulesets to nftables, just return success\n")
 {
-	g_upf->nft.mockup = true;
+	g_upf->tunmap.mockup = true;
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_nft_no_mockup, cfg_nft_no_mockup_cmd,
+DEFUN(cfg_tunmap_no_mockup, cfg_tunmap_no_mockup_cmd,
       "no mockup",
       NO_STR
       "operate nftables rulesets normally\n")
 {
-	g_upf->nft.mockup = false;
+	g_upf->tunmap.mockup = false;
 	return CMD_SUCCESS;
 }
 
-DEFUN(cfg_nft_table_name, cfg_nft_table_name_cmd,
+DEFUN(cfg_tunmap_table_name, cfg_tunmap_table_name_cmd,
       "table-name TABLE_NAME",
       "Set the nft inet table name to create and place GTP tunnel forwarding chains in"
       " (as in 'nft add table inet foo'). If multiple instances of osmo-upf are running on the same system, each"
-      " osmo-upf must have its own table name. Otherwise the names of created forwarding chains will collide.\n"
+      " osmo-upf must have its own table name. Otherwise the names of created forwarding chains will collide."
+      " The default table name is \"osmo-upf\".\n"
       "nft inet table name\n")
 {
-	osmo_talloc_replace_string(g_upf, &g_upf->nft.table_name, argv[0]);
+	osmo_talloc_replace_string(g_upf, &g_upf->tunmap.table_name, argv[0]);
+	return CMD_SUCCESS;
+}
+
+#define NFT_RULE_STR "nftables rule specifics\n"
+#define TUNMAP_STR "GTP tunmap use case (a.k.a. forwarding between two GTP tunnels)\n"
+#define TUNMAP_APPEND_STR "'tunmap append' feature is no longer available.\n"
+
+DEFUN_DEPRECATED(cfg_tunmap_nft_rule_append, cfg_tunmap_nft_rule_append_cmd,
+      "nft-rule tunmap append .NFT_RULE",
+      NFT_RULE_STR TUNMAP_STR TUNMAP_APPEND_STR TUNMAP_APPEND_STR)
+{
+	vty_out(vty, "%% deprecated config option: 'nft-rule tunmap append'%s", VTY_NEWLINE);
+	return CMD_SUCCESS;
+}
+
+DEFUN_DEPRECATED(cfg_tunmap_no_nft_rule_append, cfg_tunmap_no_nft_rule_append_cmd,
+      "no nft-rule tunmap append",
+      NO_STR NFT_RULE_STR TUNMAP_STR TUNMAP_APPEND_STR TUNMAP_APPEND_STR)
+{
+	vty_out(vty, "%% deprecated config option: 'no nft-rule tunmap append'%s", VTY_NEWLINE);
+	return CMD_SUCCESS;
+}
+
+DEFUN_DEPRECATED(show_nft_rule_append, show_nft_rule_append_cmd,
+      "show nft-rule tunmap append",
+      SHOW_STR NFT_RULE_STR TUNMAP_STR TUNMAP_APPEND_STR)
+{
+	vty_out(vty, "%% deprecated config option: 'show nft-rule tunmap append'%s", VTY_NEWLINE);
+	return CMD_SUCCESS;
+}
+
+DEFUN(show_nft_rule_tunmap_example, show_nft_rule_tunmap_example_cmd,
+      "show nft-rule tunmap example",
+      SHOW_STR NFT_RULE_STR TUNMAP_STR
+      "Print a complete nftables ruleset for a tunmap filled with example IP addresses and TEIDs\n")
+{
+	struct osmo_sockaddr_str str = {};
+	struct upf_tunmap tunmap = {
+		.access = {
+			.tun = {
+				.local.teid = 0x201,
+				.remote.teid = 0x102,
+			},
+			.chain_id = 123,
+		},
+		.core = {
+			.tun = {
+				.local.teid = 0x203,
+				.remote.teid = 0x302,
+			},
+			.chain_id = 321,
+		},
+	};
+
+	osmo_sockaddr_str_from_str2(&str, "1.1.1.1");
+	osmo_sockaddr_str_to_sockaddr(&str, &tunmap.access.tun.remote.addr.u.sas);
+
+	osmo_sockaddr_str_from_str2(&str, "2.2.2.1");
+	osmo_sockaddr_str_to_sockaddr(&str, &tunmap.access.tun.local.addr.u.sas);
+
+	osmo_sockaddr_str_from_str2(&str, "2.2.2.3");
+	osmo_sockaddr_str_to_sockaddr(&str, &tunmap.core.tun.local.addr.u.sas);
+
+	osmo_sockaddr_str_from_str2(&str, "3.3.3.3");
+	osmo_sockaddr_str_to_sockaddr(&str, &tunmap.core.tun.remote.addr.u.sas);
+
+	vty_out(vty, "%% init verdict map:%s", VTY_NEWLINE);
+	vty_out(vty, "%s%s", upf_nft_tunmap_get_table_init_str(OTC_SELECT), VTY_NEWLINE);
+	vty_out(vty, "%s%s", upf_nft_tunmap_get_vmap_init_str(OTC_SELECT), VTY_NEWLINE);
+	vty_out(vty, "%% add tunmap:%s", VTY_NEWLINE);
+	vty_out(vty, "%% %s%s", upf_nft_tunmap_to_str_c(OTC_SELECT, &tunmap), VTY_NEWLINE);
+	vty_out(vty, "%s%s", upf_nft_tunmap_get_ruleset_str(OTC_SELECT, &tunmap), VTY_NEWLINE);
+	vty_out(vty, "%% delete tunmap:%s", VTY_NEWLINE);
+	vty_out(vty, "%s%s", upf_nft_tunmap_get_ruleset_del_str(OTC_SELECT, &tunmap), VTY_NEWLINE);
+	return CMD_SUCCESS;
+}
+
+static struct cmd_node cfg_netinst_node = {
+	NETINST_NODE,
+	"%s(config-netinst)# ",
+	1,
+};
+
+DEFUN(cfg_netinst, cfg_netinst_cmd,
+      "netinst",
+      "Enter the Network Instance configuration node\n")
+{
+	vty->node = NETINST_NODE;
+	return CMD_SUCCESS;
+}
+
+static int config_write_netinst(struct vty *vty)
+{
+	vty_out(vty, "netinst%s", VTY_NEWLINE);
+	netinst_vty_write(vty, &g_upf->netinst, " ", NULL);
+	return CMD_SUCCESS;
+}
+
+DEFUN(cfg_netinst_add, cfg_netinst_add_cmd,
+      "add NAME ADDR",
+      "add Network Instance: associate a PFCP Network Instance name with a local IP address\n"
+      "Network Instance name as received in PFCP Network Instance IE\n"
+      "IP address of a local interface\n")
+{
+	const char *errmsg;
+	if (!netinst_add(g_upf, &g_upf->netinst, argv[0], argv[1], &errmsg)) {
+		vty_out(vty, "%% Error: netinst: cannot add %s %s: %s%s", argv[0], argv[1],
+			errmsg ? : "(unknown error)", VTY_NEWLINE);
+		return CMD_WARNING;
+	}
+	return CMD_SUCCESS;
+}
+
+DEFUN(show_netinst, show_netinst_cmd,
+      "show netinst [NAME]",
+      SHOW_STR "List configured Network Instance entries\n"
+      "Show the Network Instance with this name (show all when omitted)\n")
+{
+	const char *name_or_null = argc > 0 ? argv[0] : NULL;
+
+	if (!netinst_vty_write(vty, &g_upf->netinst, " ", name_or_null)) {
+		if (name_or_null)
+			vty_out(vty, "%% No such Network Instance entry%s", VTY_NEWLINE);
+		else
+			vty_out(vty, "%% No Network Instance entries configured%s", VTY_NEWLINE);
+	}
+	return CMD_SUCCESS;
+}
+
+DEFUN(cfg_netinst_clear, cfg_netinst_clear_cmd,
+      "clear",
+      "Remove all Network Instance entries\n")
+{
+	int count = netinst_clear(&g_upf->netinst);
+	vty_out(vty, "netinst entries removed: %d%s", count, VTY_NEWLINE);
 	return CMD_SUCCESS;
 }
 
@@ -282,16 +434,11 @@ DEFUN(show_pdr, show_pdr_cmd,
 DEFUN(show_gtp, show_gtp_cmd,
       "show gtp",
       SHOW_STR
-      "Active GTP tunnels and forwardings\n")
+      "Active GTP tunnels, both tunend and tunmap\n")
 {
 	struct up_peer *peer;
 	int count = 0;
 
-	if (!upf_gtp_dev_first()) {
-		vty_out(vty, "No GTP device open%s", VTY_NEWLINE);
-		return CMD_SUCCESS;
-	}
-
 	llist_for_each_entry(peer, &g_upf->pfcp.ep->peers, entry) {
 		struct up_session *session;
 		int bkt;
@@ -334,7 +481,7 @@ DEFUN(show_session, show_session_cmd,
 			}
 		}
 	}
-	vty_out(vty, "(%d fully-active + %d partially active + %d inactive)%s",
+	vty_out(vty, "(%d fully-active + %d active with some PDR/FAR ignored + %d inactive)%s",
 		fully_active_count, active_count, inactive_count, VTY_NEWLINE);
 	return CMD_SUCCESS;
 }
@@ -346,25 +493,40 @@ void upf_vty_init()
 	install_element_ve(&show_pdr_cmd);
 	install_element_ve(&show_gtp_cmd);
 	install_element_ve(&show_session_cmd);
+	install_element_ve(&show_netinst_cmd);
+	install_element_ve(&show_nft_rule_append_cmd);
 
 	install_node(&cfg_pfcp_node, config_write_pfcp);
 	install_element(CONFIG_NODE, &cfg_pfcp_cmd);
 
 	install_element(PFCP_NODE, &cfg_pfcp_local_addr_cmd);
 
-	install_node(&cfg_gtp_node, config_write_gtp);
+	install_node(&cfg_tunend_node, config_write_tunend);
+	install_element(CONFIG_NODE, &cfg_tunend_cmd);
 	install_element(CONFIG_NODE, &cfg_gtp_cmd);
 
-	install_element(GTP_NODE, &cfg_gtp_mockup_cmd);
-	install_element(GTP_NODE, &cfg_gtp_no_mockup_cmd);
-	install_element(GTP_NODE, &cfg_gtp_dev_create_cmd);
-	install_element(GTP_NODE, &cfg_gtp_dev_use_cmd);
-	install_element(GTP_NODE, &cfg_gtp_dev_del_cmd);
+	install_element(TUNEND_NODE, &cfg_tunend_mockup_cmd);
+	install_element(TUNEND_NODE, &cfg_tunend_no_mockup_cmd);
+	install_element(TUNEND_NODE, &cfg_tunend_dev_create_cmd);
+	install_element(TUNEND_NODE, &cfg_tunend_dev_use_cmd);
+	install_element(TUNEND_NODE, &cfg_tunend_dev_del_cmd);
 
-	install_node(&cfg_nft_node, config_write_nft);
+	install_node(&cfg_tunmap_node, config_write_tunmap);
+	install_element(CONFIG_NODE, &cfg_tunmap_cmd);
 	install_element(CONFIG_NODE, &cfg_nft_cmd);
 
-	install_element(NFT_NODE, &cfg_nft_mockup_cmd);
-	install_element(NFT_NODE, &cfg_nft_no_mockup_cmd);
-	install_element(NFT_NODE, &cfg_nft_table_name_cmd);
+	install_element(TUNMAP_NODE, &cfg_tunmap_mockup_cmd);
+	install_element(TUNMAP_NODE, &cfg_tunmap_no_mockup_cmd);
+	install_element(TUNMAP_NODE, &cfg_tunmap_table_name_cmd);
+	install_element(TUNMAP_NODE, &cfg_tunmap_nft_rule_append_cmd);
+	install_element(TUNMAP_NODE, &cfg_tunmap_no_nft_rule_append_cmd);
+	install_element(TUNMAP_NODE, &show_nft_rule_append_cmd);
+	install_element(TUNMAP_NODE, &show_nft_rule_tunmap_example_cmd);
+
+	install_node(&cfg_netinst_node, config_write_netinst);
+	install_element(CONFIG_NODE, &cfg_netinst_cmd);
+
+	install_element(NETINST_NODE, &cfg_netinst_clear_cmd);
+	install_element(NETINST_NODE, &cfg_netinst_add_cmd);
+	install_element(NETINST_NODE, &show_netinst_cmd);
 }

tests/Makefile.am

@@ -1,3 +1,7 @@
+SUBDIRS = \
+	unique_ids \
+	$(NULL)
+
 # The `:;' works around a Bash 3.2 bug when the output is not writeable.
 $(srcdir)/package.m4: $(top_srcdir)/configure.ac
 	:;{ \

tests/netinst.vty

@@ -0,0 +1,80 @@
+OsmoUPF> show ?
+...
+  netinst         List configured Network Instance entries
+...
+OsmoUPF> show netinst?
+  netinst  List configured Network Instance entries
+OsmoUPF> show netinst ?
+  [NAME]  Show the Network Instance with this name (show all when omitted)
+
+OsmoUPF> show netinst
+% No Network Instance entries configured
+OsmoUPF> show netinst foo
+% No such Network Instance entry
+
+OsmoUPF> enable
+
+OsmoUPF# show netinst
+% No Network Instance entries configured
+
+OsmoUPF# configure terminal
+OsmoUPF(config)# netinst
+
+OsmoUPF(config-netinst)# list
+...
+  clear
+  add NAME ADDR
+  show netinst [NAME]
+
+OsmoUPF(config-netinst)# clear?
+  clear  Remove all Network Instance entries
+OsmoUPF(config-netinst)# clear ?
+  <cr>  
+
+OsmoUPF(config-netinst)# add?
+  add  add Network Instance: associate a PFCP Network Instance name with a local IP address
+OsmoUPF(config-netinst)# add ?
+  NAME  Network Instance name as received in PFCP Network Instance IE
+OsmoUPF(config-netinst)# add foo ?
+  ADDR  IP address of a local interface
+
+OsmoUPF(config-netinst)# add foo bar
+% Error: netinst: cannot add foo bar: Network Instance address is not a valid IP address string
+OsmoUPF(config-netinst)# add foo 1.2.3.4
+OsmoUPF(config-netinst)# add foo 2.3.4.5
+% Error: netinst: cannot add foo 2.3.4.5: Network Instance entry with this name already exists
+OsmoUPF(config-netinst)# add bar 2.3.4.5
+OsmoUPF(config-netinst)# show netinst
+ add foo 1.2.3.4
+ add bar 2.3.4.5
+OsmoUPF(config-netinst)# add baz 1:2:3:4::0
+OsmoUPF(config-netinst)# show netinst
+ add foo 1.2.3.4
+ add bar 2.3.4.5
+ add baz 1:2:3:4::0
+OsmoUPF(config-netinst)# show netinst foo
+ add foo 1.2.3.4
+OsmoUPF(config-netinst)# show netinst bar
+ add bar 2.3.4.5
+OsmoUPF(config-netinst)# show netinst baz
+ add baz 1:2:3:4::0
+
+OsmoUPF(config-netinst)# show running-config
+...
+netinst
+ add foo 1.2.3.4
+ add bar 2.3.4.5
+ add baz 1:2:3:4::0
+...
+
+OsmoUPF(config-netinst)# clear
+netinst entries removed: 3
+OsmoUPF(config-netinst)# show netinst
+% No Network Instance entries configured
+OsmoUPF(config-netinst)# clear
+netinst entries removed: 0
+
+OsmoUPF(config-netinst)# show netinst?
+  netinst  List configured Network Instance entries
+OsmoUPF(config-netinst)# show netinst ?
+  [NAME]  Show the Network Instance with this name (show all when omitted)

tests/nft-rule.vty

@@ -0,0 +1,46 @@
+OsmoUPF> enable
+OsmoUPF# configure terminal
+OsmoUPF(config)# tunmap
+
+OsmoUPF(config-tunmap)# show nft-rule tunmap example
+% init verdict map:
+add table inet osmo-upf { flags owner; };
+
+add chain inet osmo-upf pre { type filter hook prerouting priority -300; policy accept; };
+add chain inet osmo-upf post { type filter hook postrouting priority 400; policy accept; };
+add map inet osmo-upf tunmap-pre { typeof ip daddr . @ih,32,32 : verdict; };
+add map inet osmo-upf tunmap-post { typeof meta mark : verdict; };
+add rule inet osmo-upf pre udp dport 2152 ip daddr . @ih,32,32 vmap @tunmap-pre;
+add rule inet osmo-upf post meta mark vmap @tunmap-post;
+
+% add tunmap:
+% ACCESS 1.1.1.1:0x102 <---> 2.2.2.1:0x201 UPF 2.2.2.3:0x203 <---> 3.3.3.3:0x302 CORE
+add chain inet osmo-upf tunmap-pre-123;
+add rule inet osmo-upf tunmap-pre-123 ip daddr set 3.3.3.3 meta mark set 123 counter accept;
+add chain inet osmo-upf tunmap-post-123;
+add rule inet osmo-upf tunmap-post-123 ip saddr set 2.2.2.3 udp sport set 2152 @ih,32,32 set 0x302 counter accept;
+add element inet osmo-upf tunmap-pre { 2.2.2.1 . 0x201 : jump tunmap-pre-123 };
+add element inet osmo-upf tunmap-post { 123 : jump tunmap-post-123 };
+add chain inet osmo-upf tunmap-pre-321;
+add rule inet osmo-upf tunmap-pre-321 ip daddr set 1.1.1.1 meta mark set 321 counter accept;
+add chain inet osmo-upf tunmap-post-321;
+add rule inet osmo-upf tunmap-post-321 ip saddr set 2.2.2.1 udp sport set 2152 @ih,32,32 set 0x102 counter accept;
+add element inet osmo-upf tunmap-pre { 2.2.2.3 . 0x203 : jump tunmap-pre-321 };
+add element inet osmo-upf tunmap-post { 321 : jump tunmap-post-321 };
+
+% delete tunmap:
+delete element inet osmo-upf tunmap-pre { 2.2.2.1 . 0x201 };
+delete element inet osmo-upf tunmap-post { 123 };
+delete chain inet osmo-upf tunmap-pre-123;
+delete chain inet osmo-upf tunmap-post-123;
+delete element inet osmo-upf tunmap-pre { 2.2.2.3 . 0x203 };
+delete element inet osmo-upf tunmap-post { 321 };
+delete chain inet osmo-upf tunmap-pre-321;
+delete chain inet osmo-upf tunmap-post-321;
+
+OsmoUPF(config-tunmap)# show nft-rule tunmap append
+% deprecated config option: 'show nft-rule tunmap append'
+OsmoUPF(config-tunmap)# nft-rule tunmap append meta nftrace set 1
+% deprecated config option: 'nft-rule tunmap append'
+OsmoUPF(config-tunmap)# no nft-rule tunmap append
+% deprecated config option: 'no nft-rule tunmap append'

tests/testsuite.at

@@ -1,2 +1,9 @@
 AT_INIT
 AT_BANNER([Regression tests.])
+
+AT_SETUP([unique_ids_test])
+AT_KEYWORDS([unique_ids_test])
+cat $abs_srcdir/unique_ids/unique_ids_test.ok > expout
+cat $abs_srcdir/unique_ids/unique_ids_test.err > experr
+AT_CHECK([$abs_top_builddir/tests/unique_ids/unique_ids_test], [], [expout], [experr])
+AT_CLEANUP

tests/unique_ids/Makefile.am

@@ -0,0 +1,41 @@
+AM_CPPFLAGS = \
+	-I$(top_srcdir)/include \
+	$(NULL)
+
+AM_CFLAGS = \
+	-Wall \
+	$(LIBOSMOCORE_CFLAGS) \
+	$(LIBOSMOVTY_CFLAGS) \
+	$(LIBOSMOCTRL_CFLAGS) \
+	$(LIBOSMOGTLV_CFLAGS) \
+	$(LIBOSMOPFCP_CFLAGS) \
+	$(LIBGTPNL_CFLAGS) \
+	$(LIBNFTNL_CFLAGS) \
+	$(LIBNFTABLES_CFLAGS) \
+	$(COVERAGE_CFLAGS) \
+	$(NULL)
+
+EXTRA_DIST = \
+	unique_ids_test.ok \
+	unique_ids_test.err \
+	$(NULL)
+
+check_PROGRAMS = \
+	unique_ids_test \
+	$(NULL)
+
+unique_ids_test_SOURCES = \
+	unique_ids_test.c \
+	$(NULL)
+
+unique_ids_test_LDADD = \
+	$(top_builddir)/src/osmo-upf/libupf.la \
+	$(NULL)
+
+unique_ids_test_LDFLAGS = \
+	-no-install \
+	$(NULL)
+
+.PHONY: update_exp
+update_exp:
+	$(builddir)/unique_ids_test >$(srcdir)/unique_ids_test.ok 2>$(srcdir)/unique_ids_test.err

tests/unique_ids/unique_ids_test.c

@@ -0,0 +1,575 @@
+/* OsmoUPF: Verify that skipping used ids works for: UP-SEID, GTP local TEID, nft ruleset chain_id. */
+
+/* (C) 2023 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ *
+ * All Rights Reserved
+ *
+ * Author: Neels Hofmeyr <nhofmeyr@sysmocom.de>
+ *
+ * SPDX-License-Identifier: GPL-2.0+
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <getopt.h>
+
+#include <nftables/libnftables.h>
+
+#include <osmocom/core/sockaddr_str.h>
+#include <osmocom/core/application.h>
+
+#include <osmocom/pfcp/pfcp_endpoint.h>
+
+#include <osmocom/upf/upf.h>
+#include <osmocom/upf/netinst.h>
+#include <osmocom/upf/up_endpoint.h>
+#include <osmocom/upf/up_peer.h>
+#include <osmocom/upf/up_session.h>
+#include <osmocom/upf/up_gtp_action.h>
+
+#define log(FMT, ARGS...) fprintf(stderr, FMT, ##ARGS)
+#define log_assert(COND) do { \
+		log("assert(" #COND ")\n"); \
+		OSMO_ASSERT(COND); \
+	} while (0)
+
+#define log_assert_expect_failure(COND) do { \
+		log("assert(" #COND ") <-- EXPECTED TO FAIL (known error)\n"); \
+		OSMO_ASSERT(!(COND)); \
+	} while (0)
+
+
+void *main_ctx;
+void *ctx;
+
+/* The override of osmo_pfcp_endpoint_tx() stores any Session Establishment Response's UP-SEID here, so that this test
+ * can reference specific sessions later.
+ */
+uint64_t last_up_seid = 0;
+
+void select_poll(void)
+{
+	while (osmo_select_main_ctx(1));
+}
+
+static void setup(const char *name)
+{
+	log("\n===== START of %s\n", name);
+	ctx = talloc_named_const(main_ctx, 0, name);
+	g_upf_alloc(ctx);
+	osmo_talloc_replace_string(g_upf, &g_upf->pfcp.vty_cfg.local_addr, "1.1.1.1");
+
+	OSMO_ASSERT(netinst_add(g_upf, &g_upf->netinst, "default", "1.1.1.1", NULL));
+
+	/* PFCP endpoint recovery timestamp overridden by time() below */
+	upf_pfcp_init();
+	/* but do not upf_pfcp_listen() */
+
+	upf_nft_init();
+
+	select_poll();
+	log("\n");
+}
+
+static void cleanup(void)
+{
+	up_endpoint_free(&g_upf->pfcp.ep);
+	upf_gtp_devs_close();
+
+	upf_gtp_genl_close();
+
+	upf_nft_free();
+
+	log("\n===== END of %s\n", talloc_get_name(ctx));
+	talloc_free(ctx);
+}
+
+static struct osmo_sockaddr *str2addr(const char *addr, uint16_t port)
+{
+	static struct osmo_sockaddr osa;
+	struct osmo_sockaddr_str str;
+	osmo_sockaddr_str_from_str(&str, addr, port);
+	osmo_sockaddr_str_to_sockaddr(&str, &osa.u.sas);
+	return &osa;
+}
+
+static struct up_peer *have_peer(const char *remote_addr, uint16_t port)
+{
+	return up_peer_find_or_add(g_upf->pfcp.ep, str2addr(remote_addr, port));
+}
+
+static struct osmo_pfcp_msg *new_pfcp_msg_for_osmo_upf_rx(struct up_peer *from_peer, enum osmo_pfcp_message_type msg_type)
+{
+	/* pfcp_endpoint discards received messages immediately after dispatching; in this test, allocate them in
+	 * OTC_SELECT so they get discarded on the next select_poll().
+	 * osmo_pfcp_msg_alloc_rx() is not useful here, it creates a blank struct to be decoded from raw data; instead,
+	 * use osmo_pfcp_msg_alloc_tx_req() which properly sets up the internal structures to match the given msg_type,
+	 * and when that is done set m->rx = true to indicate it is a message received by osmo-upf. */
+	struct osmo_pfcp_msg *m = osmo_pfcp_msg_alloc_tx_req(OTC_SELECT, &from_peer->remote_addr, msg_type);
+	m->rx = true;
+	return m;
+}
+
+static void peer_assoc(struct up_peer *peer)
+{
+	struct osmo_pfcp_msg *m = new_pfcp_msg_for_osmo_upf_rx(peer, OSMO_PFCP_MSGT_ASSOC_SETUP_REQ);
+	m->ies.assoc_setup_req.recovery_time_stamp = 1234;
+	osmo_fsm_inst_dispatch(peer->fi, UP_PEER_EV_RX_ASSOC_SETUP_REQ, m);
+	select_poll();
+}
+
+static int next_teid = 0x100;
+static int next_cp_seid = 0x100;
+
+/* Send a PFCP Session Establishment Request, and return the created session */
+static struct up_session *session_est_tunmap(struct up_peer *peer)
+{
+	struct osmo_pfcp_msg *m;
+
+	struct osmo_pfcp_ie_f_seid cp_f_seid;
+
+	struct osmo_pfcp_ie_f_teid f_teid_access_local;
+	struct osmo_pfcp_ie_outer_header_creation ohc_access;
+
+	struct osmo_pfcp_ie_f_teid f_teid_core_local;
+	struct osmo_pfcp_ie_outer_header_creation ohc_core;
+
+	struct osmo_pfcp_ie_apply_action aa = {};
+
+	osmo_pfcp_bits_set(aa.bits, OSMO_PFCP_APPLY_ACTION_FORW, true);
+
+	f_teid_access_local = (struct osmo_pfcp_ie_f_teid){
+		.choose_flag = true,
+		.choose = {
+			.ipv4_addr = true,
+		},
+	};
+
+	ohc_access = (struct osmo_pfcp_ie_outer_header_creation){
+		.teid_present = true,
+		.teid = next_teid++,
+		.ip_addr = {
+			.v4_present = true,
+			.v4 = *str2addr("5.6.7.8", 0),
+		},
+	};
+	osmo_pfcp_bits_set(ohc_access.desc_bits, OSMO_PFCP_OUTER_HEADER_CREATION_GTP_U_UDP_IPV4, true);
+
+	f_teid_core_local = (struct osmo_pfcp_ie_f_teid){
+		.choose_flag = true,
+		.choose = {
+			.ipv4_addr = true,
+		},
+	};
+
+	ohc_core = (struct osmo_pfcp_ie_outer_header_creation){
+		.teid_present = true,
+		.teid = next_teid++,
+		.ip_addr = {
+			.v4_present = true,
+			.v4 = *str2addr("13.14.15.16", 0),
+		},
+	};
+	osmo_pfcp_bits_set(ohc_core.desc_bits, OSMO_PFCP_OUTER_HEADER_CREATION_GTP_U_UDP_IPV4, true);
+
+	cp_f_seid = (struct osmo_pfcp_ie_f_seid){
+		.seid = next_cp_seid++,
+	};
+	osmo_pfcp_ip_addrs_set(&cp_f_seid.ip_addr, osmo_pfcp_endpoint_get_local_addr(g_upf->pfcp.ep->pfcp_ep));
+
+	m = new_pfcp_msg_for_osmo_upf_rx(peer, OSMO_PFCP_MSGT_SESSION_EST_REQ);
+	m->h.seid_present = true;
+	m->h.seid = 0;
+	/* GTP tunmap: remove header from both directions, and add header in both directions */
+	m->ies.session_est_req = (struct osmo_pfcp_msg_session_est_req){
+		.node_id = m->ies.session_est_req.node_id,
+		.cp_f_seid_present = true,
+		.cp_f_seid = cp_f_seid,
+		.create_pdr_count = 2,
+		.create_pdr = {
+			{
+				.pdr_id = 1,
+				.precedence = 255,
+				.pdi = {
+					.source_iface = OSMO_PFCP_SOURCE_IFACE_CORE,
+					.local_f_teid_present = true,
+					.local_f_teid = f_teid_core_local,
+				},
+				.outer_header_removal_present = true,
+				.outer_header_removal = {
+					.desc = OSMO_PFCP_OUTER_HEADER_REMOVAL_GTP_U_UDP_IPV4,
+				},
+				.far_id_present = true,
+				.far_id = 1,
+			},
+			{
+				.pdr_id = 2,
+				.precedence = 255,
+				.pdi = {
+					.source_iface = OSMO_PFCP_SOURCE_IFACE_ACCESS,
+					.local_f_teid_present = true,
+					.local_f_teid = f_teid_access_local,
+				},
+				.outer_header_removal_present = true,
+				.outer_header_removal = {
+					.desc = OSMO_PFCP_OUTER_HEADER_REMOVAL_GTP_U_UDP_IPV4,
+				},
+				.far_id_present = true,
+				.far_id = 2,
+			},
+		},
+		.create_far_count = 2,
+		.create_far = {
+			{
+				.far_id = 1,
+				.forw_params_present = true,
+				.forw_params = {
+					.destination_iface = OSMO_PFCP_DEST_IFACE_ACCESS,
+					.outer_header_creation_present = true,
+					.outer_header_creation = ohc_access,
+				},
+				.apply_action = aa,
+			},
+			{
+				.far_id = 2,
+				.forw_params_present = true,
+				.forw_params = {
+					.destination_iface = OSMO_PFCP_DEST_IFACE_CORE,
+					.outer_header_creation_present = true,
+					.outer_header_creation = ohc_core,
+				},
+				.apply_action = aa,
+			},
+		},
+	};
+
+	osmo_fsm_inst_dispatch(peer->fi, UP_PEER_EV_RX_SESSION_EST_REQ, m);
+	select_poll();
+
+	return up_session_find_by_up_seid(peer, last_up_seid);
+}
+
+static void session_del(struct up_session *session)
+{
+	struct osmo_pfcp_msg *m;
+
+	log_assert(session);
+
+	m = new_pfcp_msg_for_osmo_upf_rx(session->up_peer, OSMO_PFCP_MSGT_SESSION_DEL_REQ);
+	m->h.seid_present = true;
+	m->h.seid = session->up_seid;
+
+	osmo_fsm_inst_dispatch(session->fi, UP_SESSION_EV_RX_SESSION_DEL_REQ, m);
+	select_poll();
+}
+
+static void dump_state(void)
+{
+	struct up_peer *peer;
+	log("\n state:\n");
+	llist_for_each_entry(peer, &g_upf->pfcp.ep->peers, entry) {
+		struct up_session *session;
+		int bkt;
+		log(" | peer %s %s\n", peer->fi->name, osmo_fsm_inst_state_name(peer->fi));
+		hash_for_each(peer->sessions_by_up_seid, bkt, session, node_by_up_seid) {
+			struct up_gtp_action *a;
+			llist_for_each_entry(a, &session->active_gtp_actions, entry) {
+				if (a->kind != UP_GTP_U_TUNMAP)
+					continue;
+				log(" |  session[%s]: UP-SEID 0x%"PRIx64"; chain_id access=%u core=%u;"
+				    " local TEID access=0x%x core=0x%x\n",
+				       osmo_fsm_inst_state_name(session->fi),
+				       session->up_seid,
+				       a->tunmap.access.chain_id, a->tunmap.core.chain_id,
+				       a->tunmap.access.tun.local.teid, a->tunmap.core.tun.local.teid);
+			}
+		}
+	}
+	log("\n");
+}
+
+static void test_skip_used_id(void)
+{
+	struct up_peer *peer;
+	struct up_session *s1;
+	uint64_t s1_up_seid;
+	struct up_session *s2;
+	struct up_session *s3;
+	struct up_session *s4;
+	struct up_gtp_action *a;
+
+	setup(__func__);
+
+	log("PFCP Associate peer\n");
+	peer = have_peer("1.2.3.4", 1234);
+	peer_assoc(peer);
+	dump_state();
+
+	/* Make sure to start out all IDs with 1 */
+	g_upf->pfcp.ep->next_up_seid_state = 0;
+	g_upf->gtp.next_local_teid_state = 0;
+	g_upf->tunmap.next_chain_id_state = 0;
+
+	log("set up tunmap, which assigns first UP-SEID 0x1, local-TEID 0x1 and 0x2, chain_ids 1 and 2\n");
+	s1 = session_est_tunmap(peer);
+	dump_state();
+
+	log_assert(s1->up_seid == 1);
+	a = llist_first_entry_or_null(&s1->active_gtp_actions, struct up_gtp_action, entry);
+	log_assert(a);
+	log_assert(a->kind == UP_GTP_U_TUNMAP);
+	log_assert(a->tunmap.core.tun.local.teid == 1);
+	log_assert(a->tunmap.access.tun.local.teid == 2);
+	log_assert(a->tunmap.access.chain_id == 1);
+	log_assert(a->tunmap.core.chain_id == 2);
+	log("\n");
+
+	log("simulate wrapping of IDs back to 1\n");
+	g_upf->pfcp.ep->next_up_seid_state = 0;
+	g_upf->gtp.next_local_teid_state = 0;
+	g_upf->tunmap.next_chain_id_state = 0;
+
+	log("set up second tunmap, should use distinct IDs\n");
+	s2 = session_est_tunmap(peer);
+	dump_state();
+
+	log_assert(s2->up_seid == 2);
+	a = llist_first_entry_or_null(&s2->active_gtp_actions, struct up_gtp_action, entry);
+	log_assert(a);
+	log_assert(a->kind == UP_GTP_U_TUNMAP);
+	log_assert(a->tunmap.core.tun.local.teid == 3);
+	log_assert(a->tunmap.access.tun.local.teid == 4);
+	log_assert(a->tunmap.access.chain_id == 3);
+	log_assert(a->tunmap.core.chain_id == 4);
+	log("\n");
+
+	log("drop first tunmap (%s)\n", s1->fi->name);
+	s1_up_seid = s1->up_seid;
+	session_del(s1);
+	dump_state();
+	log_assert(up_session_find_by_up_seid(peer, s1_up_seid) == NULL);
+	log("\n");
+
+	log("again wrap all ID state back to 1\n");
+	g_upf->pfcp.ep->next_up_seid_state = 0;
+	g_upf->gtp.next_local_teid_state = 0;
+	g_upf->tunmap.next_chain_id_state = 0;
+
+	log("set up third tunmap, should now re-use same IDs as the first session\n");
+	s3 = session_est_tunmap(peer);
+	dump_state();
+
+	log_assert(s3->up_seid == 1);
+	a = llist_first_entry_or_null(&s3->active_gtp_actions, struct up_gtp_action, entry);
+	log_assert(a);
+	log_assert(a->kind == UP_GTP_U_TUNMAP);
+	log_assert(a->tunmap.core.tun.local.teid == 1);
+	log_assert(a->tunmap.access.tun.local.teid == 2);
+	log_assert(a->tunmap.access.chain_id == 1);
+	log_assert(a->tunmap.core.chain_id == 2);
+	log("\n");
+
+	log("set up 4th tunmap; chain_id state would use 3 and 4, but they are in use, so should assign 5 and 6\n");
+	s4 = session_est_tunmap(peer);
+	dump_state();
+
+	log_assert(s4->up_seid == 3);
+	a = llist_first_entry_or_null(&s4->active_gtp_actions, struct up_gtp_action, entry);
+	log_assert(a);
+	log_assert(a->kind == UP_GTP_U_TUNMAP);
+	log_assert(a->tunmap.core.tun.local.teid == 5);
+	log_assert(a->tunmap.access.tun.local.teid == 6);
+	log_assert(a->tunmap.access.chain_id == 5);
+	log_assert(a->tunmap.core.chain_id == 6);
+	log("\n");
+
+	cleanup();
+}
+
+static const struct log_info_cat test_default_categories[] = {
+	[DREF] = {
+		.name = "DREF",
+		.description = "Reference Counting",
+		.enabled = 1, .loglevel = LOGL_DEBUG,
+		.color = OSMO_LOGCOLOR_DARKGREY,
+	},
+	[DPEER] = {
+		.name = "DPEER",
+		.description = "PFCP peer association",
+		.enabled = 1, .loglevel = LOGL_DEBUG,
+		.color = OSMO_LOGCOLOR_YELLOW,
+	},
+	[DSESSION] = {
+		.name = "DSESSION",
+		.description = "PFCP sessions",
+		.enabled = 1, .loglevel = LOGL_DEBUG,
+		.color = OSMO_LOGCOLOR_BLUE,
+	},
+	[DGTP] = {
+		.name = "DGTP",
+		.description = "GTP tunneling",
+		.enabled = 1, .loglevel = LOGL_DEBUG,
+		.color = OSMO_LOGCOLOR_PURPLE,
+	},
+	[DNFT] = {
+		.name = "DNFT",
+		.description = "GTP forwarding rules via linux netfilter",
+		.enabled = 1, .loglevel = LOGL_DEBUG,
+		.color = OSMO_LOGCOLOR_PURPLE,
+	},
+};
+
+const struct log_info log_info = {
+	.cat = test_default_categories,
+	.num_cat = ARRAY_SIZE(test_default_categories),
+};
+
+static struct {
+	bool verbose;
+} cmdline_opts = {
+	.verbose = false,
+};
+
+static void print_help(const char *program)
+{
+	printf("Usage:\n"
+	       "  %s [-v]\n"
+	       "Options:\n"
+	       "  -h --help      show this text.\n"
+	       "  -v --verbose   print source file and line numbers\n",
+	       program
+	       );
+}
+
+static void handle_options(int argc, char **argv)
+{
+	while (1) {
+		int option_index = 0, c;
+		static struct option long_options[] = {
+			{"help", 0, 0, 'h'},
+			{"verbose", 1, 0, 'v'},
+			{0, 0, 0, 0}
+		};
+
+		c = getopt_long(argc, argv, "hv",
+				long_options, &option_index);
+		if (c == -1)
+			break;
+
+		switch (c) {
+		case 'h':
+			print_help(argv[0]);
+			exit(0);
+		case 'v':
+			cmdline_opts.verbose = true;
+			break;
+		default:
+			/* catch unknown options *as well as* missing arguments. */
+			fprintf(stderr, "Error in command line options. Exiting.\n");
+			exit(-1);
+			break;
+		}
+	}
+}
+
+int main(int argc, char **argv)
+{
+	handle_options(argc, argv);
+
+	main_ctx = talloc_named_const(NULL, 0, "main");
+
+	msgb_talloc_ctx_init(main_ctx, 0);
+
+	osmo_fsm_set_dealloc_ctx(OTC_SELECT);
+
+	osmo_init_logging2(main_ctx, &log_info);
+	log_set_print_category_hex(osmo_stderr_target, 0);
+	log_set_print_category(osmo_stderr_target, 1);
+	log_set_print_level(osmo_stderr_target, 1);
+	log_set_print_timestamp(osmo_stderr_target, 0);
+	log_set_print_extended_timestamp(osmo_stderr_target, 0);
+	log_set_all_filter(osmo_stderr_target, 1);
+
+	if (cmdline_opts.verbose) {
+		log_set_print_filename2(osmo_stderr_target, LOG_FILENAME_BASENAME);
+		log_set_print_filename_pos(osmo_stderr_target, LOG_FILENAME_POS_LINE_END);
+		log_set_use_color(osmo_stderr_target, 1);
+	} else {
+		log_set_print_filename2(osmo_stderr_target, LOG_FILENAME_NONE);
+		log_set_use_color(osmo_stderr_target, 0);
+	}
+
+	osmo_fsm_log_timeouts(true);
+	osmo_fsm_log_addr(false);
+
+	/* actual tests */
+	test_skip_used_id();
+
+	log_fini();
+	talloc_free(main_ctx);
+	return 0;
+}
+
+/* overrides */
+
+int osmo_pfcp_endpoint_tx(struct osmo_pfcp_endpoint *ep, struct osmo_pfcp_msg *m)
+{
+	enum osmo_pfcp_cause *cause;
+
+	log("\n[test override] PFCP tx:\n%s\n\n", osmo_pfcp_msg_to_str_c(OTC_SELECT, m));
+
+	last_up_seid = 0;
+
+	cause = osmo_pfcp_msg_cause(m);
+	switch (m->h.message_type) {
+	case OSMO_PFCP_MSGT_SESSION_EST_RESP:
+		if (*cause == OSMO_PFCP_CAUSE_REQUEST_ACCEPTED) {
+			last_up_seid = m->ies.session_est_resp.up_f_seid.seid;
+			log("osmo-upf created session 0x%"PRIx64"\n\n", last_up_seid);
+		}
+		break;
+	default:
+		break;
+	};
+	osmo_pfcp_msg_free(m);
+	return 0;
+}
+
+static void *fake_nft_ctx = (void *)0x1;
+
+struct nft_ctx *nft_ctx_new(uint32_t flags)
+{
+	log("[test override] %s()\n", __func__);
+	return fake_nft_ctx;
+}
+
+void nft_ctx_free(struct nft_ctx *ctx)
+{
+	log("[test override] %s()\n", __func__);
+	log_assert(ctx == fake_nft_ctx);
+}
+
+int nft_run_cmd_from_buffer(struct nft_ctx *nft, const char *buf)
+{
+	log("\n[test override] %s():\n%s\n", __func__, buf);
+	return 0;
+}
+
+/* for deterministic recovery_time_stamp */
+time_t time(time_t *tloc)
+{
+	log("[test override] %s()\n", __func__);
+	return 0;
+}

tests/unique_ids/unique_ids_test.err

@@ -0,0 +1,436 @@
+
+===== START of test_skip_used_id
+[test override] time()
+[test override] time()
+DLPFCP NOTICE PFCP endpoint: recovery timestamp = 0x83aa7e80 (0 seconds since UNIX epoch, which is 2208988800 seconds since NTP era 0; IETF RFC 5905)
+[test override] nft_ctx_new()
+
+[test override] nft_run_cmd_from_buffer():
+add table inet osmo-upf { flags owner; };
+
+DNFT DEBUG ran nft ruleset, 42 chars: "add table inet osmo-upf { flags owner; };\n"
+DNFT NOTICE Created nft table "osmo-upf"
+
+[test override] nft_run_cmd_from_buffer():
+add chain inet osmo-upf pre { type filter hook prerouting priority -300; policy accept; };
+add chain inet osmo-upf post { type filter hook postrouting priority 400; policy accept; };
+add map inet osmo-upf tunmap-pre { typeof ip daddr . @ih,32,32 : verdict; };
+add map inet osmo-upf tunmap-post { typeof meta mark : verdict; };
+add rule inet osmo-upf pre udp dport 2152 ip daddr . @ih,32,32 vmap @tunmap-pre;
+add rule inet osmo-upf post meta mark vmap @tunmap-post;
+
+DNFT DEBUG ran nft ruleset, 465 chars: "add chain inet osmo-upf pre { type filter hook prerouting priority -300; policy accept; };\nadd chain inet osmo-upf post { type filter hook postrouting priority 400; policy accept; };\nadd map inet osmo-upf tunmap-pre { typeof ip daddr . @ih,32,32 : verdict;..."
+
+PFCP Associate peer
+DPEER DEBUG up_peer{NOT_ASSOCIATED}: Allocated
+DPEER DEBUG up_peer(1-2-3-4){NOT_ASSOCIATED}: Updated id
+DPEER DEBUG up_peer(1-2-3-4){NOT_ASSOCIATED}: Received Event UP_PEER_EV_RX_ASSOC_SETUP_REQ
+DPEER DEBUG up_peer(1-2-3-4){NOT_ASSOCIATED}: State change to ASSOCIATED (no timeout)
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 1 (msg-tx)
+
+[test override] PFCP tx:
+PFCPv1 ASSOC_SETUP_RESP hdr={seq=0} ies={ 'Node ID'=v4:unsupported family 0 'Cause'=Request accepted (success) 'Recovery Time Stamp'=2208988800 'UP Function Features'=FTUP+BUNDL+RTTL }
+
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+DPEER NOTICE up_peer(1-2-3-4){ASSOCIATED}: Peer associated, Node-Id=v4:unsupported family 0. Local UP features: [FTUP+BUNDL+RTTL]; Peer CP features: [-]
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+
+set up tunmap, which assigns first UP-SEID 0x1, local-TEID 0x1 and 0x2, chain_ids 1 and 2
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_RX_SESSION_EST_REQ
+DSESSION DEBUG up_session(1-2-3-4){INIT}: Allocated
+DSESSION DEBUG up_session(1-2-3-4){INIT}: is child of up_peer(1-2-3-4)
+DSESSION INFO up_session(1-2-3-4){INIT}: Allocated new UP-SEID: 0x1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Updated id
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-rx: now used by 1 (msg-rx)
+DREF INFO up_session(1-2-3-4-0x1){INIT}: + msg-rx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Received Event UP_SESSION_EV_RX_SESSION_EST_REQ
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DREF DEBUG up_session(1-2-3-4-0x1){INIT}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: Allocated new local F-TEID TEID-0x1,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: New PDR-1{src:Core TEID-0x1,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x100,v4:5.6.7.8}
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: Allocated new local F-TEID TEID-0x2,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: New PDR-2{src:Access TEID-0x2,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x101,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Active PDR set:   PDR-2{src:Access TEID-0x2,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x101,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Active PDR set: + PDR-1{src:Core TEID-0x1,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x100,v4:5.6.7.8}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: GTP actions: 0 previously active; want active: 1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: want: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x100 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x101 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: enabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x100 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x101 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:847
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:847 (flush count: 1 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+add chain inet osmo-upf tunmap-pre-1;
+add rule inet osmo-upf tunmap-pre-1 ip daddr set 13.14.15.16 meta mark set 1 counter accept;
+add chain inet osmo-upf tunmap-post-1;
+add rule inet osmo-upf tunmap-post-1 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x101 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 : jump tunmap-pre-1 };
+add element inet osmo-upf tunmap-post { 1 : jump tunmap-post-1 };
+add chain inet osmo-upf tunmap-pre-2;
+add rule inet osmo-upf tunmap-pre-2 ip daddr set 5.6.7.8 meta mark set 2 counter accept;
+add chain inet osmo-upf tunmap-post-2;
+add rule inet osmo-upf tunmap-post-2 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x100 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 : jump tunmap-pre-2 };
+add element inet osmo-upf tunmap-post { 2 : jump tunmap-post-2 };
+
+
+DNFT DEBUG ran nft ruleset, 847 chars: "add chain inet osmo-upf tunmap-pre-1;\nadd rule inet osmo-upf tunmap-pre-1 ip daddr set 13.14.15.16 meta mark set 1 counter accept;\nadd chain inet osmo-upf tunmap-post-1;\nadd rule inet osmo-upf tunmap-post-1 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x100 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x101 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1: Enabled tunmap, nft chain IDs: access--1-> <-2--core
+
+[test override] PFCP tx:
+PFCPv1 SESSION_EST_RESP hdr={seq=0 SEID=0x100} ies={ 'Node ID'=v4:unsupported family 0 'Cause'=Request accepted (success) 'F-SEID'=0x1,v4:1.1.1.1 'Created PDR'={ { 'PDR ID'=1 'F-TEID'=TEID-0x1,v4:1.1.1.1 }, { 'PDR ID'=2 'F-TEID'=TEID-0x2,v4:1.1.1.1 } } }
+
+osmo-upf created session 0x1
+
+DREF DEBUG up_session(1-2-3-4-0x1){INIT}: - msg-tx: now used by 1 (msg-rx)
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: State change to ESTABLISHED (no timeout)
+DSESSION NOTICE up_session(1-2-3-4-0x1){ESTABLISHED}: Session established: peer:1.2.3.4 SEID-r:0x100 SEID-l:0x1 state:ESTABLISHED PDR-active:2/2 FAR-active:2/2 GTP-active:1
+DREF INFO up_session(1-2-3-4-0x1){ESTABLISHED}: - msg-rx: now used by 0 (-)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Received Event UP_SESSION_EV_USE_COUNT_ZERO
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-rx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+ |  session[ESTABLISHED]: UP-SEID 0x1; chain_id access=1 core=2; local TEID access=0x2 core=0x1
+
+assert(s1->up_seid == 1)
+assert(a)
+assert(a->kind == UP_GTP_U_TUNMAP)
+assert(a->tunmap.core.tun.local.teid == 1)
+assert(a->tunmap.access.tun.local.teid == 2)
+assert(a->tunmap.access.chain_id == 1)
+assert(a->tunmap.core.chain_id == 2)
+
+simulate wrapping of IDs back to 1
+set up second tunmap, should use distinct IDs
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_RX_SESSION_EST_REQ
+DSESSION DEBUG up_session(1-2-3-4){INIT}: Allocated
+DSESSION DEBUG up_session(1-2-3-4){INIT}: is child of up_peer(1-2-3-4)
+DSESSION INFO up_session(1-2-3-4){INIT}: Allocated new UP-SEID: 0x2
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: Updated id
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-rx: now used by 1 (msg-rx)
+DREF INFO up_session(1-2-3-4-0x2){INIT}: + msg-rx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: Received Event UP_SESSION_EV_RX_SESSION_EST_REQ
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DREF DEBUG up_session(1-2-3-4-0x2){INIT}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DSESSION INFO up_session(1-2-3-4-0x2){INIT}: Allocated new local F-TEID TEID-0x3,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x2){INIT}: New PDR-1{src:Core TEID-0x3,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x102,v4:5.6.7.8}
+DSESSION INFO up_session(1-2-3-4-0x2){INIT}: Allocated new local F-TEID TEID-0x4,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x2){INIT}: New PDR-2{src:Access TEID-0x4,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x103,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: Active PDR set:   PDR-2{src:Access TEID-0x4,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x103,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: Active PDR set: + PDR-1{src:Core TEID-0x3,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x102,v4:5.6.7.8}
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: GTP actions: 0 previously active; want active: 1
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: want: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: enabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:847
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:847 (flush count: 2 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+add chain inet osmo-upf tunmap-pre-3;
+add rule inet osmo-upf tunmap-pre-3 ip daddr set 13.14.15.16 meta mark set 3 counter accept;
+add chain inet osmo-upf tunmap-post-3;
+add rule inet osmo-upf tunmap-post-3 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x103 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x4 : jump tunmap-pre-3 };
+add element inet osmo-upf tunmap-post { 3 : jump tunmap-post-3 };
+add chain inet osmo-upf tunmap-pre-4;
+add rule inet osmo-upf tunmap-pre-4 ip daddr set 5.6.7.8 meta mark set 4 counter accept;
+add chain inet osmo-upf tunmap-post-4;
+add rule inet osmo-upf tunmap-post-4 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x102 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x3 : jump tunmap-pre-4 };
+add element inet osmo-upf tunmap-post { 4 : jump tunmap-post-4 };
+
+
+DNFT DEBUG ran nft ruleset, 847 chars: "add chain inet osmo-upf tunmap-pre-3;\nadd rule inet osmo-upf tunmap-pre-3 ip daddr set 13.14.15.16 meta mark set 3 counter accept;\nadd chain inet osmo-upf tunmap-post-3;\nadd rule inet osmo-upf tunmap-post-3 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1: Enabled tunmap, nft chain IDs: access--3-> <-4--core
+
+[test override] PFCP tx:
+PFCPv1 SESSION_EST_RESP hdr={seq=0 SEID=0x101} ies={ 'Node ID'=v4:unsupported family 0 'Cause'=Request accepted (success) 'F-SEID'=0x2,v4:1.1.1.1 'Created PDR'={ { 'PDR ID'=1 'F-TEID'=TEID-0x3,v4:1.1.1.1 }, { 'PDR ID'=2 'F-TEID'=TEID-0x4,v4:1.1.1.1 } } }
+
+osmo-upf created session 0x2
+
+DREF DEBUG up_session(1-2-3-4-0x2){INIT}: - msg-tx: now used by 1 (msg-rx)
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x2){INIT}: State change to ESTABLISHED (no timeout)
+DSESSION NOTICE up_session(1-2-3-4-0x2){ESTABLISHED}: Session established: peer:1.2.3.4 SEID-r:0x101 SEID-l:0x2 state:ESTABLISHED PDR-active:2/2 FAR-active:2/2 GTP-active:1
+DREF INFO up_session(1-2-3-4-0x2){ESTABLISHED}: - msg-rx: now used by 0 (-)
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: Received Event UP_SESSION_EV_USE_COUNT_ZERO
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-rx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+ |  session[ESTABLISHED]: UP-SEID 0x1; chain_id access=1 core=2; local TEID access=0x2 core=0x1
+ |  session[ESTABLISHED]: UP-SEID 0x2; chain_id access=3 core=4; local TEID access=0x4 core=0x3
+
+assert(s2->up_seid == 2)
+assert(a)
+assert(a->kind == UP_GTP_U_TUNMAP)
+assert(a->tunmap.core.tun.local.teid == 3)
+assert(a->tunmap.access.tun.local.teid == 4)
+assert(a->tunmap.access.chain_id == 3)
+assert(a->tunmap.core.chain_id == 4)
+
+drop first tunmap (up_session(1-2-3-4-0x1))
+assert(session)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Received Event UP_SESSION_EV_RX_SESSION_DEL_REQ
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 1 (msg-tx)
+DREF INFO up_session(1-2-3-4-0x1){ESTABLISHED}: + msg-tx: now used by 1 (msg-tx)
+
+[test override] PFCP tx:
+PFCPv1 SESSION_DEL_RESP hdr={seq=0 SEID=0x100} ies={ 'Cause'=Request accepted (success) }
+
+DREF INFO up_session(1-2-3-4-0x1){ESTABLISHED}: - msg-tx: now used by 0 (-)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Received Event UP_SESSION_EV_USE_COUNT_ZERO
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+DSESSION NOTICE up_session(1-2-3-4-0x1){ESTABLISHED}: Session releasing: peer:1.2.3.4 SEID-r:0x100 SEID-l:0x1 state:ESTABLISHED PDR-active:2/2 FAR-active:2/2 GTP-active:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:381
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:381 (flush count: 3 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 };
+delete element inet osmo-upf tunmap-post { 1 };
+delete chain inet osmo-upf tunmap-pre-1;
+delete chain inet osmo-upf tunmap-post-1;
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 };
+delete element inet osmo-upf tunmap-post { 2 };
+delete chain inet osmo-upf tunmap-pre-2;
+delete chain inet osmo-upf tunmap-post-2;
+
+
+DNFT DEBUG ran nft ruleset, 381 chars: "delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 };\ndelete element inet osmo-upf tunmap-post { 1 };\ndelete chain inet osmo-upf tunmap-pre-1;\ndelete chain inet osmo-upf tunmap-post-1;\ndelete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 };\ndelete ..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x100 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x101 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1: Disabled tunmap, nft chain IDs: access--1-> <-2--core
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: State change to WAIT_USE_COUNT (no timeout)
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: GTP actions: 0 previously active; want active: 0
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: Terminating (cause = OSMO_FSM_TERM_REGULAR)
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: Removing from parent up_peer(1-2-3-4)
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: GTP actions: 0 previously active; want active: 0
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: Freeing instance
+DSESSION DEBUG up_session(1-2-3-4-0x1){WAIT_USE_COUNT}: Deallocated
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_SESSION_TERM
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+ |  session[ESTABLISHED]: UP-SEID 0x2; chain_id access=3 core=4; local TEID access=0x4 core=0x3
+
+assert(up_session_find_by_up_seid(peer, s1_up_seid) == NULL)
+
+again wrap all ID state back to 1
+set up third tunmap, should now re-use same IDs as the first session
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_RX_SESSION_EST_REQ
+DSESSION DEBUG up_session(1-2-3-4){INIT}: Allocated
+DSESSION DEBUG up_session(1-2-3-4){INIT}: is child of up_peer(1-2-3-4)
+DSESSION INFO up_session(1-2-3-4){INIT}: Allocated new UP-SEID: 0x1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Updated id
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-rx: now used by 1 (msg-rx)
+DREF INFO up_session(1-2-3-4-0x1){INIT}: + msg-rx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Received Event UP_SESSION_EV_RX_SESSION_EST_REQ
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DREF DEBUG up_session(1-2-3-4-0x1){INIT}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: Allocated new local F-TEID TEID-0x1,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: New PDR-1{src:Core TEID-0x1,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x104,v4:5.6.7.8}
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: Allocated new local F-TEID TEID-0x2,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x1){INIT}: New PDR-2{src:Access TEID-0x2,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x105,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Active PDR set:   PDR-2{src:Access TEID-0x2,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x105,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: Active PDR set: + PDR-1{src:Core TEID-0x1,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x104,v4:5.6.7.8}
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: GTP actions: 0 previously active; want active: 1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: want: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: enabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:847
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:847 (flush count: 4 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+add chain inet osmo-upf tunmap-pre-1;
+add rule inet osmo-upf tunmap-pre-1 ip daddr set 13.14.15.16 meta mark set 1 counter accept;
+add chain inet osmo-upf tunmap-post-1;
+add rule inet osmo-upf tunmap-post-1 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x105 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 : jump tunmap-pre-1 };
+add element inet osmo-upf tunmap-post { 1 : jump tunmap-post-1 };
+add chain inet osmo-upf tunmap-pre-2;
+add rule inet osmo-upf tunmap-pre-2 ip daddr set 5.6.7.8 meta mark set 2 counter accept;
+add chain inet osmo-upf tunmap-post-2;
+add rule inet osmo-upf tunmap-post-2 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x104 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 : jump tunmap-pre-2 };
+add element inet osmo-upf tunmap-post { 2 : jump tunmap-post-2 };
+
+
+DNFT DEBUG ran nft ruleset, 847 chars: "add chain inet osmo-upf tunmap-pre-1;\nadd rule inet osmo-upf tunmap-pre-1 ip daddr set 13.14.15.16 meta mark set 1 counter accept;\nadd chain inet osmo-upf tunmap-post-1;\nadd rule inet osmo-upf tunmap-post-1 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1: Enabled tunmap, nft chain IDs: access--1-> <-2--core
+
+[test override] PFCP tx:
+PFCPv1 SESSION_EST_RESP hdr={seq=0 SEID=0x102} ies={ 'Node ID'=v4:unsupported family 0 'Cause'=Request accepted (success) 'F-SEID'=0x1,v4:1.1.1.1 'Created PDR'={ { 'PDR ID'=1 'F-TEID'=TEID-0x1,v4:1.1.1.1 }, { 'PDR ID'=2 'F-TEID'=TEID-0x2,v4:1.1.1.1 } } }
+
+osmo-upf created session 0x1
+
+DREF DEBUG up_session(1-2-3-4-0x1){INIT}: - msg-tx: now used by 1 (msg-rx)
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x1){INIT}: State change to ESTABLISHED (no timeout)
+DSESSION NOTICE up_session(1-2-3-4-0x1){ESTABLISHED}: Session established: peer:1.2.3.4 SEID-r:0x102 SEID-l:0x1 state:ESTABLISHED PDR-active:2/2 FAR-active:2/2 GTP-active:1
+DREF INFO up_session(1-2-3-4-0x1){ESTABLISHED}: - msg-rx: now used by 0 (-)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Received Event UP_SESSION_EV_USE_COUNT_ZERO
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-rx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+ |  session[ESTABLISHED]: UP-SEID 0x1; chain_id access=1 core=2; local TEID access=0x2 core=0x1
+ |  session[ESTABLISHED]: UP-SEID 0x2; chain_id access=3 core=4; local TEID access=0x4 core=0x3
+
+assert(s3->up_seid == 1)
+assert(a)
+assert(a->kind == UP_GTP_U_TUNMAP)
+assert(a->tunmap.core.tun.local.teid == 1)
+assert(a->tunmap.access.tun.local.teid == 2)
+assert(a->tunmap.access.chain_id == 1)
+assert(a->tunmap.core.chain_id == 2)
+
+set up 4th tunmap; chain_id state would use 3 and 4, but they are in use, so should assign 5 and 6
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_RX_SESSION_EST_REQ
+DSESSION DEBUG up_session(1-2-3-4){INIT}: Allocated
+DSESSION DEBUG up_session(1-2-3-4){INIT}: is child of up_peer(1-2-3-4)
+DSESSION INFO up_session(1-2-3-4){INIT}: Allocated new UP-SEID: 0x3
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: Updated id
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: + msg-rx: now used by 1 (msg-rx)
+DREF INFO up_session(1-2-3-4-0x3){INIT}: + msg-rx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: Received Event UP_SESSION_EV_RX_SESSION_EST_REQ
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DREF DEBUG up_session(1-2-3-4-0x3){INIT}: + msg-tx: now used by 2 (msg-rx,msg-tx)
+DSESSION INFO up_session(1-2-3-4-0x3){INIT}: Allocated new local F-TEID TEID-0x5,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x3){INIT}: New PDR-1{src:Core TEID-0x5,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x106,v4:5.6.7.8}
+DSESSION INFO up_session(1-2-3-4-0x3){INIT}: Allocated new local F-TEID TEID-0x6,v4:1.1.1.1
+DSESSION INFO up_session(1-2-3-4-0x3){INIT}: New PDR-2{src:Access TEID-0x6,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x107,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: Active PDR set:   PDR-2{src:Access TEID-0x6,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-2{FORW dst:Core,GTP_U_UDP_IPV4,TEID:0x107,v4:13.14.15.16}
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: Active PDR set: + PDR-1{src:Core TEID-0x5,v4:1.1.1.1 decaps-GTP_U_UDP_IPV4} --> FAR-1{FORW dst:Access,GTP_U_UDP_IPV4,TEID:0x106,v4:5.6.7.8}
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: GTP actions: 0 previously active; want active: 1
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: want: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: enabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:847
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:847 (flush count: 5 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+add chain inet osmo-upf tunmap-pre-5;
+add rule inet osmo-upf tunmap-pre-5 ip daddr set 13.14.15.16 meta mark set 5 counter accept;
+add chain inet osmo-upf tunmap-post-5;
+add rule inet osmo-upf tunmap-post-5 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x107 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x6 : jump tunmap-pre-5 };
+add element inet osmo-upf tunmap-post { 5 : jump tunmap-post-5 };
+add chain inet osmo-upf tunmap-pre-6;
+add rule inet osmo-upf tunmap-pre-6 ip daddr set 5.6.7.8 meta mark set 6 counter accept;
+add chain inet osmo-upf tunmap-post-6;
+add rule inet osmo-upf tunmap-post-6 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32 set 0x106 counter accept;
+add element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x5 : jump tunmap-pre-6 };
+add element inet osmo-upf tunmap-post { 6 : jump tunmap-post-6 };
+
+
+DNFT DEBUG ran nft ruleset, 847 chars: "add chain inet osmo-upf tunmap-pre-5;\nadd rule inet osmo-upf tunmap-pre-5 ip daddr set 13.14.15.16 meta mark set 5 counter accept;\nadd chain inet osmo-upf tunmap-post-5;\nadd rule inet osmo-upf tunmap-post-5 ip saddr set 1.1.1.1 udp sport set 2152 @ih,32,32..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1: Enabled tunmap, nft chain IDs: access--5-> <-6--core
+
+[test override] PFCP tx:
+PFCPv1 SESSION_EST_RESP hdr={seq=0 SEID=0x103} ies={ 'Node ID'=v4:unsupported family 0 'Cause'=Request accepted (success) 'F-SEID'=0x3,v4:1.1.1.1 'Created PDR'={ { 'PDR ID'=1 'F-TEID'=TEID-0x5,v4:1.1.1.1 }, { 'PDR ID'=2 'F-TEID'=TEID-0x6,v4:1.1.1.1 } } }
+
+osmo-upf created session 0x3
+
+DREF DEBUG up_session(1-2-3-4-0x3){INIT}: - msg-tx: now used by 1 (msg-rx)
+DREF DEBUG up_peer(1-2-3-4){ASSOCIATED}: - msg-tx: now used by 1 (msg-rx)
+DSESSION DEBUG up_session(1-2-3-4-0x3){INIT}: State change to ESTABLISHED (no timeout)
+DSESSION NOTICE up_session(1-2-3-4-0x3){ESTABLISHED}: Session established: peer:1.2.3.4 SEID-r:0x103 SEID-l:0x3 state:ESTABLISHED PDR-active:2/2 FAR-active:2/2 GTP-active:1
+DREF INFO up_session(1-2-3-4-0x3){ESTABLISHED}: - msg-rx: now used by 0 (-)
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: Received Event UP_SESSION_EV_USE_COUNT_ZERO
+DREF INFO up_peer(1-2-3-4){ASSOCIATED}: - msg-rx: now used by 0 (-)
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Received Event UP_PEER_EV_USE_COUNT_ZERO
+
+ state:
+ | peer up_peer(1-2-3-4) ASSOCIATED
+ |  session[ESTABLISHED]: UP-SEID 0x3; chain_id access=5 core=6; local TEID access=0x6 core=0x5
+ |  session[ESTABLISHED]: UP-SEID 0x1; chain_id access=1 core=2; local TEID access=0x2 core=0x1
+ |  session[ESTABLISHED]: UP-SEID 0x2; chain_id access=3 core=4; local TEID access=0x4 core=0x3
+
+assert(s4->up_seid == 3)
+assert(a)
+assert(a->kind == UP_GTP_U_TUNMAP)
+assert(a->tunmap.core.tun.local.teid == 5)
+assert(a->tunmap.access.tun.local.teid == 6)
+assert(a->tunmap.access.chain_id == 5)
+assert(a->tunmap.core.chain_id == 6)
+
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Terminating (cause = OSMO_FSM_TERM_REGULAR)
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: Terminating (cause = OSMO_FSM_TERM_PARENT)
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: Removing from parent up_peer(1-2-3-4)
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: GTP actions: 1 previously active; want active: 0
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: active: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: disabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:381
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:381 (flush count: 6 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x6 };
+delete element inet osmo-upf tunmap-post { 5 };
+delete chain inet osmo-upf tunmap-pre-5;
+delete chain inet osmo-upf tunmap-post-5;
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x5 };
+delete element inet osmo-upf tunmap-post { 6 };
+delete chain inet osmo-upf tunmap-pre-6;
+delete chain inet osmo-upf tunmap-post-6;
+
+
+DNFT DEBUG ran nft ruleset, 381 chars: "delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x6 };\ndelete element inet osmo-upf tunmap-post { 5 };\ndelete chain inet osmo-upf tunmap-pre-5;\ndelete chain inet osmo-upf tunmap-post-5;\ndelete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x5 };\ndelete ..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x106 GTP-access-l:1.1.1.1 TEID-access-l:0x6 GTP-core-r:13.14.15.16 TEID-core-r:0x107 GTP-core-l:1.1.1.1 TEID-core-l:0x5 PFCP-peer:1.2.3.4 SEID-l:0x3 PDR-access:2 PDR-core:1: Disabled tunmap, nft chain IDs: access--5-> <-6--core
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: Freeing instance
+DSESSION DEBUG up_session(1-2-3-4-0x3){ESTABLISHED}: Deallocated
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Terminating (cause = OSMO_FSM_TERM_PARENT)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Removing from parent up_peer(1-2-3-4)
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: GTP actions: 1 previously active; want active: 0
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: active: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: disabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:381
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:381 (flush count: 7 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 };
+delete element inet osmo-upf tunmap-post { 1 };
+delete chain inet osmo-upf tunmap-pre-1;
+delete chain inet osmo-upf tunmap-post-1;
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 };
+delete element inet osmo-upf tunmap-post { 2 };
+delete chain inet osmo-upf tunmap-pre-2;
+delete chain inet osmo-upf tunmap-post-2;
+
+
+DNFT DEBUG ran nft ruleset, 381 chars: "delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x2 };\ndelete element inet osmo-upf tunmap-post { 1 };\ndelete chain inet osmo-upf tunmap-pre-1;\ndelete chain inet osmo-upf tunmap-post-1;\ndelete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x1 };\ndelete ..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x104 GTP-access-l:1.1.1.1 TEID-access-l:0x2 GTP-core-r:13.14.15.16 TEID-core-r:0x105 GTP-core-l:1.1.1.1 TEID-core-l:0x1 PFCP-peer:1.2.3.4 SEID-l:0x1 PDR-access:2 PDR-core:1: Disabled tunmap, nft chain IDs: access--1-> <-2--core
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Freeing instance
+DSESSION DEBUG up_session(1-2-3-4-0x1){ESTABLISHED}: Deallocated
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: Terminating (cause = OSMO_FSM_TERM_PARENT)
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: Removing from parent up_peer(1-2-3-4)
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: GTP actions: 1 previously active; want active: 0
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: active: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: disabling: GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1
+DNFT INFO Added NFT ruleset to queue: n:1 strlen:381
+DNFT INFO Flushing NFT ruleset queue: reached max nr of rules: n:1 strlen:381 (flush count: 8 avg rules per flush: 1)
+
+[test override] nft_run_cmd_from_buffer():
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x4 };
+delete element inet osmo-upf tunmap-post { 3 };
+delete chain inet osmo-upf tunmap-pre-3;
+delete chain inet osmo-upf tunmap-post-3;
+delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x3 };
+delete element inet osmo-upf tunmap-post { 4 };
+delete chain inet osmo-upf tunmap-pre-4;
+delete chain inet osmo-upf tunmap-post-4;
+
+
+DNFT DEBUG ran nft ruleset, 381 chars: "delete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x4 };\ndelete element inet osmo-upf tunmap-post { 3 };\ndelete chain inet osmo-upf tunmap-pre-3;\ndelete chain inet osmo-upf tunmap-post-3;\ndelete element inet osmo-upf tunmap-pre { 1.1.1.1 . 0x3 };\ndelete ..."
+DGTP NOTICE GTP:tunmap GTP-access-r:5.6.7.8 TEID-access-r:0x102 GTP-access-l:1.1.1.1 TEID-access-l:0x4 GTP-core-r:13.14.15.16 TEID-core-r:0x103 GTP-core-l:1.1.1.1 TEID-core-l:0x3 PFCP-peer:1.2.3.4 SEID-l:0x2 PDR-access:2 PDR-core:1: Disabled tunmap, nft chain IDs: access--3-> <-4--core
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: Freeing instance
+DSESSION DEBUG up_session(1-2-3-4-0x2){ESTABLISHED}: Deallocated
+DPEER NOTICE up_peer(1-2-3-4){ASSOCIATED}: Peer removed
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Freeing instance
+DPEER DEBUG up_peer(1-2-3-4){ASSOCIATED}: Deallocated
+[test override] nft_ctx_free()
+assert(ctx == fake_nft_ctx)
+
+===== END of test_skip_used_id

tests/upf.vty

@@ -13,23 +13,86 @@ OsmoUPF(config-pfcp)# local-addr ?
   IP_ADDR  IP address
 OsmoUPF(config-pfcp)# exit
 
+OsmoUPF(config)# # ensure its old name "gtp" enters the tunend node
 OsmoUPF(config)# gtp
-OsmoUPF(config-gtp)# list
+OsmoUPF(config-tunend)# list
 ...
+  mockup
+  no mockup
   dev create DEVNAME [LISTEN_ADDR]
-  dev use DEVNAME
+  dev use DEVNAME [LOCAL_ADDR]
   dev delete DEVNAME
 
-OsmoUPF(config-gtp)# dev?
+OsmoUPF(config-tunend)# exit
+OsmoUPF(config)# tunend
+OsmoUPF(config-tunend)# list
+...
+  dev create DEVNAME [LISTEN_ADDR]
+  dev use DEVNAME [LOCAL_ADDR]
+  dev delete DEVNAME
+
+OsmoUPF(config-tunend)# dev?
   dev  Configure the GTP device to use for encaps/decaps.
-OsmoUPF(config-gtp)# dev ?
-  create  Add GTP device, creating a new Linux kernel GTP device. Will listen on GTPv1 port 2152 and GTPv0 port 3386 on the specified interface, or on ANY if LISTEN_ADDR is omitted.
+OsmoUPF(config-tunend)# dev ?
+  create  Add GTP device, creating a new Linux kernel GTP device. Will listen on GTPv1 port 2152 and GTPv0 port 3386 on the specified LISTEN_ADDR
   use     Add GTP device, using an existing Linux kernel GTP device, e.g. created by 'gtp-link'
   delete  Remove a GTP device from the configuration, and delete the Linux kernel GTP device if it was created here.
-OsmoUPF(config-gtp)# dev create ?
+OsmoUPF(config-tunend)# dev create ?
   DEVNAME  device name, e.g. 'apn0'
-OsmoUPF(config-gtp)# dev create foo ?
-  [LISTEN_ADDR]  IPv4 or IPv6 address to listen on, omit for ANY
-OsmoUPF(config-gtp)# dev delete ?
+OsmoUPF(config-tunend)# dev create foo ?
+  [LISTEN_ADDR]  IPv4 or IPv6 address to listen on, omit for ANY. LISTEN_ADDR is used to pick a GTP device matching the local address for a PFCP Network Instance, which are configured in the 'netinst' node.
+OsmoUPF(config-tunend)# dev use ?
   DEVNAME  device name, e.g. 'apn0'
-OsmoUPF(config-gtp)# exit
+OsmoUPF(config-tunend)# dev use foo ?
+  [LOCAL_ADDR]  The local GTP address this device listens on. It is assumed to be ANY when omitted. LOCAL_ADDR is used to pick a GTP device matching the local address for a PFCP Network Instance, which are configured in the 'netinst' node.
+OsmoUPF(config-tunend)# dev delete ?
+  DEVNAME  device name, e.g. 'apn0'
+OsmoUPF(config-tunend)# exit
+
+OsmoUPF(config)# # ensure its old name "nft" enters the tunmap node
+OsmoUPF(config)# nft
+OsmoUPF(config-tunmap)# list
+...
+  mockup
+  no mockup
+  table-name TABLE_NAME
+  show nft-rule tunmap example
+OsmoUPF(config-tunmap)# exit
+
+OsmoUPF(config)# tunmap
+OsmoUPF(config-tunmap)# list
+...
+  mockup
+  no mockup
+  table-name TABLE_NAME
+  show nft-rule tunmap example
+
+OsmoUPF(config-tunmap)# mockup?
+  mockup  don't actually send rulesets to nftables, just return success
+OsmoUPF(config-tunmap)# no ?
+  mockup  operate nftables rulesets normally
+
+OsmoUPF(config-tunmap)# table-name?
+  table-name  Set the nft inet table name to create and place GTP tunnel forwarding chains in (as in 'nft add table inet foo'). If multiple instances of osmo-upf are running on the same system, each osmo-upf must have its own table name. Otherwise the names of created forwarding chains will collide. The default table name is "osmo-upf".
+OsmoUPF(config-tunmap)# table-name ?
+  TABLE_NAME  nft inet table name
+
+OsmoUPF(config-tunmap)# nft-rule?
+% There is no matched command.
+OsmoUPF(config-tunmap)# nft-rule ?
+% There is no matched command.
+OsmoUPF(config-tunmap)# nft-rule tunmap ?
+% There is no matched command.
+OsmoUPF(config-tunmap)# nft-rule tunmap append ?
+% There is no matched command.
+
+OsmoUPF(config-tunmap)# show?
+  show  Show running system information
+OsmoUPF(config-tunmap)# show ?
+...
+  nft-rule        nftables rule specifics
+...
+OsmoUPF(config-tunmap)# show nft-rule ?
+  tunmap  GTP tunmap use case (a.k.a. forwarding between two GTP tunnels)
+OsmoUPF(config-tunmap)# show nft-rule tunmap ?
+  example  Print a complete nftables ruleset for a tunmap filled with example IP addresses and TEIDs