35 Commits

Author SHA1 Message Date
Louis
bd9d794a2c feat: add string renderer (#277) 2024-01-19 21:28:14 -08:00
Vincent Bernat
1a900554fb producer: increase performance of sampling rate lookup (#264)
Formatting a string with version and domain when looking up sampling
rate hurts performance. Instead, use a dedicated structure for key. It
also reduces the pressure on the GC.
2023-12-29 03:29:09 -08:00
Louis
3f017c46b0 bugfix: check etype size before decoding (#252) 2023-12-12 18:44:59 -08:00
Louis
da044ac77a producer: add datetime render (#225) 2023-12-06 05:03:45 -08:00
Louis
5ba02103eb bug: fix kafka transport default partitioning (#246) 2023-12-06 04:53:51 -08:00
Louis
34a0c1618e decoders: port reader changes from v1 to v2 (#204) 2023-09-04 03:19:41 -07:00
Louis
cfed6da9c1 producer: fix fragment for NetFlow (#215) 2023-09-01 15:22:47 -07:00
Louis
9f2b0806f0 producer: fix fragment (#214) 2023-09-01 14:41:55 -07:00
Louis
bfb23ba283 producer: various improvements (#213)
* feat: add IpFlags field in the protobuf (sFlow and IPFIX populate it)
* fix: fragments parsing from previous commit
* refactor: sflow parsing functions
* feat: decode IPv6 fragment headers
* tests: add producer tests
2023-09-01 14:18:18 -07:00
Louis
8988f2c244 producer: add missing renderer for bgp_next_hop (#210) 2023-08-31 22:31:34 -07:00
Paweł Mieczkowski
026fef546e fix: missing time received ns in netflow v5 (#209)
Co-authored-by: Paweł Mieczkowski <pawel.mieczkowski@redge.com>
2023-08-26 08:52:20 -07:00
Louis
a6fe3ffb12 producer: bugfix of sampling mapping for ipfix (#207) 2023-08-23 20:51:54 -07:00
Louis
458e90777f producer: fix fragments (#196) 2023-08-09 19:57:45 -07:00
Louis
ae56e41786 refactor: v2 (#150) 2023-08-09 19:47:20 -07:00
Louis
d59f209302 mapping: support endianness (#130) 2023-03-28 22:05:47 -07:00
Louis
40e5ef08fd mapping: allow lists to be mapped (#149) 2023-03-28 21:55:21 -07:00
Louis
332655476b proto: change protobuf definition case (#109) 2022-10-08 15:47:31 -07:00
Louis
e3471e58c6 fix single ip template case (#106) 2022-10-08 15:16:40 -07:00
Vincent Bernat
57fad2e0c8 Fix SrcAS/DstAS for sFlow (#125) 2022-10-08 08:41:47 -07:00
Louis
1468ff77c2 BGP Next Hop, Communities and AS-Path fields (#101)
* Add BGP Next Hop field to flow message (#11)
* Add BGP Communities and AS Path fields (#12)

Co-authored-by: Simon Podlipsky <simon@podlipsky.net>
2022-07-20 22:04:43 -07:00
Louis
e0abb7ba07 Add MPLS decoding for IPFIX (#95)
* Add MPLS decoding for IPFIX
Co-authored-by: David Roy <door7302@gmail.com>
2022-07-20 21:40:11 -07:00
Louis
0020934574 Observation domain (#98)
* netflow: Add observation domain and point to message

The ObservationDomainID and ObservationPointID are two IPFIX fields that
identify the entity that is capturing flows and can be used to enrich
the context around a specific sample.

Parse these fields from the sample and add them to the FlowMessage.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Co-authored-by: Adrian Moreno <amorenoz@redhat.com>
2022-07-20 21:27:51 -07:00
Ian Pye
97d58ccfe2 Adding field names to allow ipfix yaml mapping field to parse 2022-07-06 13:53:57 -07:00
Joel Takvorian
10dc7271d5 In IPFIX DeltaMicroseconds case, basetime must be converted to MS 2022-05-09 11:05:49 +02:00
Alejandro Antonio Moreno Sancho
23e75e85f8 Added TimeFlowStartMs and TimeFlowEndMs to Netflow v10 (IPFIX) 2022-05-07 12:11:47 +02:00
Alejandro Antonio Moreno Sancho
3e6a0510fd Added TimeFlowStartMs and TimeFlowEndMs to Netflow v5 2022-05-07 12:05:42 +02:00
Alejandro Antonio Moreno Sancho
55bef5b2f0 Added TimeFlowStartMs and TimeFlowEndMs to Netflow v9 2022-05-07 12:00:33 +02:00
Mario Macias
615b9f697c Avoid unnecessary memory allocations 2022-01-26 17:24:35 +01:00
lspgn
27863691f0 Bugfix: index out of range in some cases 2022-01-06 21:59:02 +01:00
Louis
b0b73b2b90 Merge pull request #38 from vincentbernat/fix/defer-unlock
style: defer unlock when possible/not trivial
2021-09-23 20:44:05 -07:00
Louis
536b08812f Custom map flow fields (#36)
* adds dataframe link decoding
* can map NetFlow/IPFIX fields and bytes sections from sFlow/packets to any field inside the protobuf
* add CLI argument for loading a mapping yaml file
2021-09-23 20:41:17 -07:00
Vincent Bernat
95945d3042 style: defer unlock when possible/not trivial
Defer unlocking just after taking a lock when possible (when unlock is
done at the very end) and when not trivial (the function body is more
than a couple of lines). This simplifies a bit some functions (no need
to unlock before each return) and for the other, it may avoid a bug in
the future in case a return is inserted into the body of a function.

Use of defer has been optimized a lot in Go and it is believed that
simpler defers have zero overhead since Go 1.14:
https://golang.org/doc/go1.14#runtime

> This release improves the performance of most uses of defer to incur
> almost zero overhead compared to calling the deferred function
> directly. As a result, defer can now be used in performance-critical
> code without overhead concerns.
2021-09-23 10:01:24 +02:00
Adrian Moreno
9247511c44 ipfix: support relative flow timestamps
IPFIX supports sending flowEndDeltaMicroseconds (159) and
flowEndDeltaMicroseconds (160) to provide flow timestamps relative to
the exportTime in the IPFIX Message Header.

Use them to calculate flow TimeFlowStart and TimeFlowEnd.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
2021-09-08 17:12:31 +02:00
lspgn
04c8543306 Bugfix NetFlow/IPFIX producer
* Source and destination mac addresses are fed from `NFV9_FIELD_xx_yyy_MAC`
  Fixes a bug where the samples were missing either source or destination Mac.
  This is supposed to be linked to the sampling direction.
2021-06-09 18:08:55 -07:00
lspgn
e4bacbc4bd init 2021-05-25 17:33:26 -07:00