176 Commits

Author SHA1 Message Date
Louis
0020934574 Observation domain (#98)
* netflow: Add observation domain and point to message

The ObservationDomainID and ObservationPointID are two IPFIX fields that
identify the entity that is capturing flows and can be used to enrich
the context around a specific sample.

Parse these fields from the sample and add them to the FlowMessage.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Co-authored-by: Adrian Moreno <amorenoz@redhat.com>
2022-07-20 21:27:51 -07:00
Louis
838505b10b Merge pull request #86 from OlivierCazade/log-transport-error
Add logging of transport error
2022-07-06 20:24:53 -07:00
Louis
1834503bb5 Merge pull request #90 from kentik/fix-yaml-mapping
Adding field names to allow ipfix yaml mapping field to parse
2022-07-06 20:23:31 -07:00
Ian Pye
97d58ccfe2 Adding field names to allow ipfix yaml mapping field to parse 2022-07-06 13:53:57 -07:00
Olivier Cazade
78f2ce561d Add logging of transport error 2022-06-13 17:36:28 +02:00
Louis
5300494e47 Merge pull request #82 from jotak/fix-deltams-convert
In IPFIX DeltaMicroseconds case, basetime must be converted to MS
2022-05-09 08:52:30 -07:00
Joel Takvorian
10dc7271d5 In IPFIX DeltaMicroseconds case, basetime must be converted to MS 2022-05-09 11:05:49 +02:00
Louis
0c878cde53 Merge pull request #81 from aams-telefonica/feature/timeflow-start-end-ms
Add TimeFlowStartMs and TimeFlowEndMs to Netflow v5, v9, v10 (IPFIX)
2022-05-07 10:20:25 -07:00
Alejandro Antonio Moreno Sancho
23e75e85f8 Added TimeFlowStartMs and TimeFlowEndMs to Netflow v10 (IPFIX) 2022-05-07 12:11:47 +02:00
Alejandro Antonio Moreno Sancho
3e6a0510fd Added TimeFlowStartMs and TimeFlowEndMs to Netflow v5 2022-05-07 12:05:42 +02:00
Alejandro Antonio Moreno Sancho
55bef5b2f0 Added TimeFlowStartMs and TimeFlowEndMs to Netflow v9 2022-05-07 12:00:33 +02:00
Louis
d59dad6db5 Merge pull request #77 from netsampler/bug/netflowv9-variable
support netflow v9 variable length
2022-04-28 20:53:15 -07:00
lspgn
99e086f684 support netflow v9 variable length 2022-04-02 10:40:35 -07:00
Louis
58f0f97a62 Merge pull request #72 from vincentbernat/feature/netflow-tests
Add some tests for Netflow decoding
2022-03-13 22:10:46 -07:00
Louis
bd480ebebb Merge pull request #75 from vincentbernat/fix/counter-type
Fix type for a counter when decoding a template flowset
2022-03-03 11:03:25 -08:00
Vincent Bernat
a5fc8ecfb6 Fix type for a counter when decoding a template flowset 2022-03-03 19:45:34 +01:00
Vincent Bernat
e3f8f990ec Add some tests for Netflow decoding
The tests are a bit more expansive than the existing tests for sFlow
or NFv5 as we check the whole structure. I am also testing the
String() function as it is easier to read. It is a bit redundant, but
checking only for the wire format makes it difficult to compare with
Wireshark. Only testing for the textual representation is not totally
good as it is not what is used by users of the decode function.
2022-02-26 16:47:08 +01:00
Louis
7d4c41a161 Merge pull request #71 from mariomac/rmalloc
Avoid unnecessary memory allocations
2022-02-08 20:39:55 -08:00
Mario Macias
615b9f697c Avoid unnecessary memory allocations 2022-01-26 17:24:35 +01:00
Louis
8d59905c44 Merge pull request #68 from netsampler/bugfix/enricher-2
Bugfix: issues when reading a partial chunk of protobuf from stdin
2022-01-24 20:34:11 -08:00
lspgn
7acb84835b Bugfix: issues when reading a partial chunk of protobuf from stdin 2022-01-24 20:30:30 -08:00
Louis
20e8e56709 Merge pull request #65 from netsampler/bugfix/offset
Bugfix: index out of range in some cases
2022-01-06 22:00:10 +01:00
lspgn
27863691f0 Bugfix: index out of range in some cases 2022-01-06 21:59:02 +01:00
Louis
4308c483a4 Merge pull request #61 from netsampler/feature/upgrade-sarama
upgrade sarama to 1.30.1
2021-12-27 14:20:55 +01:00
lspgn
05b436277b upgrade sarama to 1.30.1 2021-12-27 14:17:43 +01:00
Louis
a8a50f785c Merge pull request #60 from netsampler/feature/add-sctp
Map ProtoName for SCTP
2021-12-20 17:06:40 +01:00
lspgn
def05e071b Map ProtoName for SCTP 2021-12-20 16:53:00 +01:00
Louis
2e1cf5bbfd Merge pull request #48 from netsampler/feature/kafka-flush-control
Add flags to control Kafka Flush parameters
2021-11-13 16:41:38 -08:00
Louis
de5e751729 Merge pull request #53 from netsampler/bugfix/optiontemplates
Bugfix: decoding OptionsTemplateSet
2021-11-13 16:41:32 -08:00
lspgn
43cf8b58d9 Bugfix: decoding OptionsTemplateSet
* Was improperly decoding scope fields
* Start removing custom errors structures (will optimize template not found)
2021-11-13 15:49:27 -08:00
Louis
bf66556023 Merge pull request #52 from vincentbernat/fix/non-terminated-stream
utils: copy payload before accepting another UDP packet
2021-11-10 08:10:39 -08:00
Vincent Bernat
0f5528170d utils: copy payload before accepting another UDP packet
In d1e1ace318 ("Allow Flow Routines to be cancellable (#40)"), the
payload was passed to another goroutine and erased by the next packet
to be received if the goroutine did not process it fast enough. Make
a copy before passing it to the goroutine to fix that.
2021-11-10 15:56:49 +01:00
Louis
8fd64006ef Merge pull request #51 from leoluk/remove-kafkastate
Remove unused KafkaState
2021-11-05 09:16:34 -07:00
Leopold Schabel
56a2ffa46b Remove unused KafkaState 2021-11-03 16:59:57 +01:00
Mario Macias
d1e1ace318 Allow Flow Routines to be cancellable (#40)
* Allow Flow Routines to be cancellable
2021-10-31 16:42:07 -07:00
lspgn
c145be66cd Add flags to control Kafka Flush parameters 2021-10-31 10:23:03 -07:00
Louis
92043a6233 Merge pull request #47 from netsampler/cleanup/mod-extension
cleanup: bump to go 1.17, sarama to 1.30.0, fix extensions
2021-10-30 19:51:12 -07:00
lspgn
ec08b786c8 cleanup: bump to go 1.17, sarama to 1.30.0, fix extensions 2021-10-30 19:47:11 -07:00
Louis
7baa828267 Merge pull request #45 from shyam334/update-dep-sarama
deps: bump sarama to v1.29.1
2021-10-30 19:40:58 -07:00
Shyam Sundar
a9f9289f6b deps: bump sarama to v1.29.1 2021-10-28 15:06:42 +11:00
Vincent Bernat
17a96d9911 netflow: correctly decode options template set (#39)
netflow: correctly decode options template set
2021-09-23 20:46:39 -07:00
Louis
b0b73b2b90 Merge pull request #38 from vincentbernat/fix/defer-unlock
style: defer unlock when possible/not trivial
2021-09-23 20:44:05 -07:00
Louis
536b08812f Custom map flow fields (#36)
* adds dataframe link decoding
* can map NetFlow/IPFIX fields and bytes sections from sFlow/packets to any field inside the protobuf
* add CLI argument for loading a mapping yaml file
2021-09-23 20:41:17 -07:00
Vincent Bernat
95945d3042 style: defer unlock when possible/not trivial
Defer unlocking just after taking a lock when possible (when unlock is
done at the very end) and when not trivial (the function body is more
than a couple of lines). This simplifies a bit some functions (no need
to unlock before each return) and for the other, it may avoid a bug in
the future in case a return is inserted into the body of a function.

Use of defer has been optimized a lot in Go and it is believed that
simpler defers have zero overhead since Go 1.14:
https://golang.org/doc/go1.14#runtime

> This release improves the performance of most uses of defer to incur
> almost zero overhead compared to calling the deferred function
> directly. As a result, defer can now be used in performance-critical
> code without overhead concerns.
2021-09-23 10:01:24 +02:00
Louis
defd786b2a Merge pull request #34 from jotak/podman
Makefile: parameterized DOCKER_REPO and DOCKER_BIN
2021-09-09 21:51:39 -07:00
Louis
13daf3b669 Merge pull request #33 from amorenoz/delta
ipfix: support relative flow timestamps
2021-09-09 21:51:12 -07:00
Joel Takvorian
d3e3fdb0f8 Makefile: parameterized DOCKER_REPO and DOCKER_BIN
e.g. allowing to build with podman

Also add gitignore for dist/

Signed-off-by: Joel Takvorian <jtakvori@redhat.com>
2021-09-09 14:53:26 +02:00
Adrian Moreno
9247511c44 ipfix: support relative flow timestamps
IPFIX supports sending flowEndDeltaMicroseconds (159) and
flowEndDeltaMicroseconds (160) to provide flow timestamps relative to
the exportTime in the IPFIX Message Header.

Use them to calculate flow TimeFlowStart and TimeFlowEnd.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
2021-09-08 17:12:31 +02:00
Louis
11acc67852 Merge pull request #32 from loganmc10/patch-1
Fix README for SrcPort and DstPort
2021-09-07 08:43:42 -07:00
Logan
5ad65d5100 Fix README for SrcPort and DstPort 2021-09-07 07:35:09 -06:00