88 Commits

Author SHA1 Message Date
Ryan Smith
153191f6c5 feat: add proxy protocol support
This change adds a new proxyproto package to support Proxy Protocol versions 1 and 2. This package allows extraction of the original source IP address from Proxy Protocol headers.
2025-05-13 09:55:54 -07:00
Ryan Smith
c83ebcc342 chore: revise ordering of log data
This change moves the `remote_ip` log field after source IP data when a proxy header is configured.
2025-05-13 07:05:51 -07:00
Ryan Smith
a9dcc759f7 build: update modules 2025-05-08 16:35:03 -07:00
Ryan Smith
f9d7b767bc refactor: switch from net.IP to netip.Addr
This change switches net.IP to netip.Addr added in Go 1.18. This results in slightly better performance and memory utilization for very large threat feeds (over 500,000 entries).
2025-05-08 16:26:33 -07:00
Ryan Smith
375da6eeac feat: log custom header as source IP if set
This change updates the logging behavior of the HTTP honeypot. If a custom custom source IP header is configured:
- The actual connecting IP is logged as `remote_ip`.
- The IP extracted from the header is logged as `source_ip`.
- Any problems extracting an IP from the header results in `source_ip` falling back to the actual connecting IP.
- A new `source_ip_parsed` field indicates whether an IP was extrracted from the header.
- If parsing fails, a `source_ip_error` field is included with the error message.

If no custom header is configured, logging behavior remains unchanged.

This change improves usability of the threat feed web interface when you have HTTP honeypots behind a proxy. By logging the original client IP as `source_ip`, the application now correctly displays the actual source of the connection, rather than your proxy's IP address.
2025-05-08 13:45:58 -07:00
Ryan Smith
dc06d64b5b build: prevent -dirty tag in Docker builds
Ensure Docker builds don't add a `-dirty` suffix to the version number.

When building for Docker, `make` is called which in turn calls `git describe` to get the version number. Sometimes this would append `-dirty` even when clean.

This change adds `git update-index -q --refresh` before calling `make` to ensure the version tag is returned correctly.
2025-04-16 15:34:47 -07:00
Ryan Smith
41345f04bd build: add Linux ARM64 build target 2025-04-16 10:34:48 -07:00
Ryan Smith
c0e6010143 build: replace amd64 with x64 in binary names
This change replaces 'amd64' with 'x64' in the file names of the output binaries.

The binary names are now set directly in each build target, rather than declared as variables.
2025-04-16 10:33:36 -07:00
Ryan Smith
2736c20158 feat(threatfeed): display TLS configuration 2025-04-16 08:51:08 -07:00
Ryan Smith
8ebec3a8c4 feat: add TLS support to threat feed server
This change adds optional support for running the threat feed server over HTTPS. This is controlled via the configuration file. Depending on the confgiuration, the threat feed may operate over either HTTP or HTTPS, but not both.

The following configuration options are added to the threat feed (the `<threatFeed>` section in the conffguration file):
- `<enableTLS>` - If `true`, the threat feed uses TLS. If `false` or if this is missing, use HTTP.
- `<certPath>` - Path to TLS cert in PEM format.
- `<keyPath>` - Path to private key in PEM format.

Default configuration files are updated to include the new settings. The TLS feature is off by default. Existing user configuration files only need to be updated if this feature is needed. Otherwise, existing configuration files start the threat feed using HTTP as before.

When the threat feed server starts in TLS mode, it automatically generates a self-signed cert if the cert and key files aen't found.
2025-04-16 08:33:36 -07:00
Ryan Smith
650489bd5c feat: add fixed delay to basic auth 2025-04-16 07:44:24 -07:00
Ryan Smith
da42f21f75 refactor: move cert generator to separate package 2025-04-16 07:43:15 -07:00
Ryan Smith
0a4d4536ba chore: revise error strings and comments 2025-04-16 07:35:58 -07:00
Ryan Smith
148d99876f build: update modules 2025-04-16 07:24:18 -07:00
Ryan Smith
90fbc24479 feat: controlled error responses for HTTP honeypot
Add `withCustomError` middleware that intercepts HTTP error responses and replaces them with a custom error response.

This is used when the HTTP honeypot is configured to serve content from a directory. It ensures that all error responses from http.FileServerFS are controlled and predicatable.
2025-04-15 14:44:26 -07:00
Ryan Smith
60fe095dff feat: disable directory listings when serving custom content
Add noDirectoryFS wrapper to disable directory listings from http.FileServerFS. This is used when the HTTP honeypot is configured to serve custom content from a specified directory.
2025-04-15 14:39:07 -07:00
Ryan Smith
40dbc05d6f feat: serve content from a directory in HTTP honeypot
Add support for serving static files from a directory specified via the existing `homePagePath` setting. When this setting points to a directory, the honeypot serves files rooted at that directory. The original behavior of serving a single file is still supported.

When serving from a directory, the honeypot may serve files from the directory root and from any subdirectories. Symbolic links are followed, provided they don't lead outside the specified root directory.

Main changes:
- Add `responseMode` type to represent how the honeypot serves content (built-in default, specific file, files from a directory).
- Add `responseConfig` struct to store the responseMode and related configuration.
- Add `determineConfig` function to construct a responseConfig when the honeypot starts.
- Update the honeypot request handler to serve content based on the response mode.
- Add `serveErrorPage` function to serve error responses as needed.
2025-04-15 12:48:58 -07:00
Ryan Smith
9e14d3886a Merge pull request #1 from eltociear/patch-1
docs: update README.md
2025-04-10 07:10:47 -07:00
Ikko Eltociear Ashimine
abaa098099 docs: update README.md
honeyot  -> honeypot
2025-04-10 14:51:00 +09:00
Ryan Smith
62b166c62a Update README.md
Update threat feed example output to show 'observations' count rather than 'threat_score'. The threat score feature was replaced with observation count in a previous change.
2025-04-08 11:10:18 -07:00
Ryan Smith
53fd03cd46 Revise screenshot 2025-04-08 10:37:21 -07:00
Ryan Smith
a1dfb7f648 threatfeed: Pre-parse and cache html templates
This change pre-parses all html templates from the `templates` directory and stores the results globally. With this change, http handlers no longer need to re-parse templates on every request.
2025-04-07 16:57:09 -07:00
Ryan Smith
540b0b940c threatfeed: Add honeypot log data statistics
This change adds the ability to view various statistics for honeypot log data. This includes views such as unique SSH usernames, unique HTTP paths, unique HTTP host headers, etc.

A new `/logs/{logtype}/{subtype}` route is added for rendering stats.
2025-04-07 16:40:18 -07:00
Ryan Smith
7bc73f6695 threatfeed: move nav bar to dedicated template
This change moves the nav bar for the threat feed web interface to a dedicated template defined in `nav.html`. HTTP handlers and existing HTML templates are updated to utilize the new template.
2025-04-06 22:44:27 -07:00
Ryan Smith
d0f046593e theatfeed: tooltips in live feed + style updates
This change adds tooltips to the live logs for HTTP log data. Hovering over an HTTP even reveals the full HTTP request details.

Updated style.css to support the new tooltips.

Other changes:
- Minor color changes applied to the web feed.
- Minor text revisions in `home.html` and `docs.html`.
2025-04-06 20:15:54 -07:00
Ryan Smith
444a446b0f webfeed: format dates and numbers via javascript
This change adjusts the webfeed.html template to return timestamps in ISO 8601 format in UTC and instead uses JavaScript to format and display using the user's local time.

JavaScript is also used to add a thousands seprator to values in the 'Observations' column.

When formatting the 'Added' column, the time is dropped and now displays as YYYY-MM-DD.
2025-04-06 14:55:18 -07:00
Ryan Smith
0462ed7b4c Explicitly ignore errors in WS handlers 2025-04-05 14:27:00 -07:00
Ryan Smith
ecbe1d4972 Append -dirty or -broken tag to version if local changes found 2025-04-05 14:26:10 -07:00
Ryan Smith
4eebe8029f Revise screenshots 2025-04-04 14:51:40 -07:00
Ryan Smith
0e66c52a16 Update README.md 2025-04-04 09:32:54 -07:00
Ryan Smith
7334aac745 Update screenshots 2025-04-04 08:30:35 -07:00
Ryan Smith
fd60dc89eb Add ability to monitor honeypot logs in realtime via WebSockets
This change adds support for WebSockets using Google's WebSocket package.

When the threat feed server is starting, a Go function is created to monitor honeypot log data via a channel. When log data is received on the channel, it is broadcast to all connected WebSocket clients.

A /live endpoint and handler is added for serving the live.html template. This page displays the log data in real time by utilizing WebSockets.

Updated the nav bar on all html pages to include the new 'Live' icon for accessing the realtime log.
2025-04-03 14:07:50 -07:00
Ryan Smith
35c0eb06f8 Move log-related handlers to separate file 2025-04-03 10:52:05 -07:00
Ryan Smith
d3f7cb4e86 Add logmonitor to configuration 2025-03-31 08:59:49 -07:00
Ryan Smith
c3ca87c7af Add logmonitor package for monitoring writes to log files
The Monitor type is an io.Writer that sends data to a channel. It is meant for use with the honeypot logging system. This will allow the threat feed to provide real-time monitoring of the logging system while simultaneously logging the data to disk.
2025-03-29 12:18:05 -07:00
Ryan Smith
6ba9f0acf5 Add ability to view the running configuration
This change adds a /config handler for displaying the Deceptifeed configuration. An icon is added to the nav bar for accessing the page.

- Add config.html template for displaying the Deceptifeed configuration.
- Add config.html supporting styles to style.css.
- Add /config http handler to render the config.html template.
- Add icon to nav bar for accessing /config page.
2025-03-27 13:26:52 -07:00
Ryan Smith
94dce2c13a Revise styles 2025-03-27 10:03:47 -07:00
Ryan Smith
4fd048c287 Remove feature for custom threat file
This change removes the `CustomThreatsPath` setting from the threat feed configuration. The default configuration files are updated with this setting removed.
2025-03-26 18:23:37 -07:00
Ryan Smith
7bad11a4a7 Allow comments in exclude list
This change allows for comments in the exclude list using the `#` symbol. The `#` symbol on a line and everything after it is ignored when parsing the exclude list.
2025-03-26 18:17:31 -07:00
Ryan Smith
30c3095541 Remove v prefix from version tag 2025-03-26 18:13:18 -07:00
Ryan Smith
920759db70 Explicitly disable threat feed for UDP honeypots
This change explicitly disables the threat feed for UDP honeypots in the configuration. The UDP honeypot server does not implement the threat feed.
2025-03-24 11:17:05 -07:00
Ryan Smith
7dc7b1ee83 Add setting and getting version information
- Add `Version` string var to config package for storing version information.
- Update Makefile to set the `Version` variable at build time using the latest Git tag.
- Add `-version` flag to main package to print version information and exit.
- Remove setting the GO111MODULE environment variable from Makefile when building. It's not needed.
2025-03-22 08:56:10 -07:00
Ryan Smith
f6cd4c783e Change nav bar names and icons
This change renames `Web Feed` to `Threats` and `Honeypot Logs` to `Logs` in the navigation bar.

Apply visual tweaks to the nav bar.
2025-03-20 16:34:48 -07:00
Ryan Smith
4cf8d15402 Add FilePath field to Config struct
The FilePath field stores the absolute path to the running configuration file and is set while the configuration file is initially loading.
2025-03-20 09:44:40 -07:00
Ryan Smith
f5d6f9f78b Sort server configuration by port number
This change sorts the honeypot server configuration by port number during application initialization.

This is purely cosmetic and intended for a planned config viewer feature.
2025-03-20 09:39:07 -07:00
Ryan Smith
60ab753c42 Remove threat score feature, replace with observation count
This change removes the 'threat score' feature which allowed users to configure each honeypot server with a variable 'score' when updating the threat feed.

It is replaced with a fixed observation count that is incremented by 1 for each honeypot interaction.

The field `threat_score` has been replaced with `observations` in all API call parameters and threat feed data.

The `threat_score` field in the CSV file has been renamed to `observations`. Existing threat feed CSV files will be automatically updated on the next threat feed save.
2025-03-20 09:20:15 -07:00
Ryan Smith
b23e9b4a9e Remove minimum threat score feature and settings 2025-03-19 20:50:21 -07:00
Ryan Smith
f72cf4ddba Use centralized stylesheet
This change removes the CSS sections from HTML templates and switches to using a single stylesheet at /css/style.css.
2025-03-19 19:55:14 -07:00
Ryan Smith
d50bce3fbf Explicitly set default SSH banner
This change explicitly sets the default SSH banner in the running configuration when the application starts.

If starting without a configuration file, the global default banner is always used.

If starting with a configuration file, the global default banner is used when no banner is specified.
2025-03-19 19:36:59 -07:00
Ryan Smith
5b7618ad5e Fix log path not set when using CLI flags
This resolves a bug introduced in commit bc7fcef4b5 which resulted in log paths not being set when the app was launched without a configuration file.

This change corrects the problem by explicitly setting the log path for each honeypot server when no configuration file is provided.
2025-03-18 11:39:24 -07:00
Ryan Smith
764188cf2b Require private IP when accessing log data 2025-03-18 09:40:58 -07:00
Ryan Smith
00b747341b Display message when feed/logs are empty
Adjust the html templates for web feed and log viewers to check if there's any data to display. If no data, a message is shown informing the user that there's no data.
2025-03-18 07:36:01 -07:00
Ryan Smith
97cddb8cfe Ensure log files are read in a consistent order
Use a slice instead of a map to track unique paths to ensure log files are read in the correct order.
2025-03-18 07:26:27 -07:00
Ryan Smith
9384834da1 Remove click effect on logo in nav bar 2025-03-17 19:56:10 -07:00
Ryan Smith
0e0a18e1f1 Adjust Makefile to strip debug info and add architectures
- Strip debug information to reduce binary size, using `-ldflags="-s -w"`.
- Remove file system paths from the build environment in the resulting binary, using `-trimpath`.
- Add targets to build for Linux, FreeBSD, and Windows.
2025-03-17 17:43:36 -07:00
Ryan Smith
115efa5b69 Change output directory to ./bin when compiling
Previously, binaries were compiled to ./out.
2025-03-17 16:14:33 -07:00
Ryan Smith
5dc1a6d91c Change output directory to ./bin when compiling
Previously, binaries were compiled to ./out.
2025-03-17 15:54:14 -07:00
Ryan Smith
0d09a59d3c Explicitly ignore errors from rand.Read
The error is ignored because rand.Read is guranteed to never return an error.
2025-03-17 14:55:06 -07:00
Ryan Smith
86eb9b773a Update dependencies 2025-03-17 14:26:04 -07:00
Ryan Smith
182262d474 Add ability to view honeypot logs from the threat feed server 2025-03-17 14:08:13 -07:00
Ryan Smith
bc7fcef4b5 Refactor log path initialization
Previously:
When no log path was specified for a server, it would fall back to the global/default log path during logger initialization. However, the `LogPath` field didn't update and would not reflect the actual path used by the logger.

Now:
Log path determination is handled while the configuration is loading. If a server falls back to the default log path, the `LogPath` field is updated to reflect the actual path used by the logger.
2025-03-17 13:25:09 -07:00
Ryan Smith
857966808c Revise CSS styling for threat feed server 2025-03-14 10:42:17 -07:00
Ryan Smith
70e8180b2b Add custom 404 page 2025-03-09 12:35:10 -07:00
Ryan Smith
1bde74187f Add nav bar and visual tweaks to management pages
- Add a horizontal navigation bar to the top of all management pages.
- Tweaks to colors and styling of management pages.
- Rename the endpoint '/threatfeed' to '/webfeed'.
- Rename the file 'htmlfeed.html' to 'webfeed.html'.
2025-03-09 10:08:05 -07:00
Ryan Smith
96b5be5758 Minor visual and structural revisions
- Remove the `/feed` endpoint, as this is already handled by `/plain`.
- Rename `/html` endpoint to `/threatfeed`.
- Add a `/docs` endpoint for displaying information on accessing the threat feed.
- Make the Deceptifeed logo smaller on HTML pages.
- Revise the layout of the initial homepage. Endpoint docs are now moved and linked to /docs.
- Remove `/html` from the list of threat feed endpoints in the docs, as it doesn't belong here (there is a link to the html threat feed on the home page).
2025-03-07 13:34:41 -08:00
Ryan Smith
3e72919170 Use slog.DiscardHandler when logging is disabled
This change switches to using slog.DiscardHandler, added to Go 1.24, when logging is disabled.
2025-03-03 22:26:41 -08:00
Ryan Smith
122e1ca83d Update rand.Read error handling
As of Go 1.24, rand.Read is guaranteed to never return an error. This change removes the error check and the associated fallback function when calling rand.Read.
2025-03-03 22:07:45 -08:00
Ryan Smith
c96313242a Update dependencies 2025-03-03 21:59:24 -08:00
Ryan Smith
bfc121ce06 Increment minimum Go version to v1.24 2025-03-03 21:49:30 -08:00
Ryan Smith
99b9760830 Update threat feed server to receive full app config
This change updates the threat feed server so that the entire application config is passed in to the threat feed server. This allows the threat feed server to access its own configuration as well as the settings for each honeypot server and any other application-wide settings.
2024-12-30 10:39:01 -08:00
Ryan Smith
126577d842 Revise saveCSV routine
Refactor saveCSV to use a bufio.NewWriter and manually format CSV data rather than using the built-in CSV writer. This results in slightly better performance and memory usage when the threat feed is large (>500k entries).

Various miscellaneous changes:
- Rename `mutex` variable to `mu`.
- Lock the mutex in `loadCSV` because it accesses the iocData map.
2024-12-28 23:35:14 -08:00
Ryan Smith
6f4d1d9921 Revise error messages 2024-12-27 22:04:13 -08:00
Ryan Smith
849709ae01 Explicitly ignore errors from file.Close()
In logrotate.OpenFile(), explicitly ignore errors from file.Close().
2024-12-27 21:43:48 -08:00
Ryan Smith
ef2bc057f4 Update dependencies 2024-12-27 21:17:36 -08:00
Ryan Smith
ae596b82e8 Automatically rotate honeypot logs
This change configures the honeypot loggers to use the internal `logrotate` package. This enables automatic log file rotation. The maximum log size is set to 50 MB.
2024-12-27 21:06:59 -08:00
Ryan Smith
646c09a4fa Add logrotate package to manage log file rotation
The logrotate package implements an io.WriteCloser with automatic file rotation. When the file size exceeds a specified limit, the file is renamed with a `.1` suffix. Subsequent rotations overwrite the file with the `.1` suffix, meaning only one backup is retained. This approach keeps the rotation simple.
2024-12-27 17:31:21 -08:00
Ryan Smith
0269fe34d2 Ensure only 1 file handle/logger is created per log file
This change adjusts logger initialization to ensure only 1 file handle and 1 logger is created per unique log path.

Each honeypot server may have its own log path specified, which may be unique or may be shared with other honeypot servers.

Previously, each honeypot server would open a separate file handle and logger, even if the file was already opened by another server.
2024-12-27 09:01:02 -08:00
Ryan Smith
563c76696b Change default SSH banner to appear as OpenSSH 9.6
This change adjusts the default SSH server version string to `SSH-2.0-OpenSSH_9.6`. This makes SSH honeypots appear as OpenSSH 9.6.

The change is applied both to default configuration files and as an application default when no configuration is provided.
2024-12-27 08:48:43 -08:00
Ryan Smith
1a631e7e14 Use quoted strings when printing certain fields
This change adjusts the SSH, TCP, and UDP honeypots to print quoted (escaped) strings to the terminal for certain log fields rather than raw strings. The adjusted fields are SSH username, SSH password, TCP responses, and UDP received data.
2024-12-26 11:29:06 -08:00
Ryan Smith
079becbd82 Update threat feed html templates
Changes to HTML feed template:
- Drop milliseconds from added and last seen dates.
- Make the logo a link to return to the threat feed homepage.
- Fix logo sizing on small screens.
- Reduce logo margins.

Changes to home page template:
- Update /stix API endpoint documentation to change references from /stix/indicators and /stix/observables to just /stix.
- Update TAXII documentation to include the new sightings collection.
- Update TAXII documentation with shortened collection names.
- Set `break-word` wrapping on code snippets to fix rendering on small screens.
- Collapse API endpoint table on small screens.
- Fix logo sizing on small screens.
- Reduce logo margins.
2024-12-26 10:27:46 -08:00
Ryan Smith
b47d5278f4 Consilidate STIX endpoints to /stix
This change removes the /stix/indicators and /stix/observables endpoints and replaces them with a single /stix endpoint. The /stix endpoint returns the threat feed as STIX indicators.
2024-12-26 10:16:04 -08:00
Ryan Smith
505e1fa2e0 Add sightings TAXII collection
- Add `sightings` TAXII collection to represent IP addresses observed interacting with Deceptifeed as STIX sightings
- Rename collection aliases `deceptifeed-indicators` and `deceptifeed-observables` to `indicators` and `observables`.
- Add `convertToSightings` method for converting the threat feed to STIX sightings.
- Set a confidence score of 100 to STIX indicators.
- Rename label `honeypot` to `honeypot-interaction` on STIX indicators.
2024-12-26 10:05:55 -08:00
Ryan Smith
45bb7e48b9 Define a maximum threat score of 999,999,999
This change sets a maximum threat score of 999,999,999. Previously, the maximum threat score was the max signed int value (9,223,372,036,854,775,807 on 64-bit systems).
2024-12-26 09:42:54 -08:00
Ryan Smith
73e2dd1c4b Add Sighting STIX Relationship Object
This change adds a `Sighting` struct to the stix package for representing a `Sighting` STIX Relationship Object (SRO).
2024-12-26 08:49:17 -08:00
Ryan Smith
1df1a045d0 Fix: Don't log "User-Agent" in headers key
This commit ensures the `User-Agent` value is removed from the HTTP request headers when logging the request. `User-Agent` is logged in the `event_details` array, so it should not duplicate the value in the `headers` array when logging. This commit fixes the issue introduced in commit 12ada38faa which normalised header names to lowercase.
2024-12-19 16:34:57 -08:00
Ryan Smith
41eab266fa Fix missing end tag in /html template 2024-12-17 10:36:56 -08:00
Ryan Smith
c35c8ebda9 Revise HTML styles 2024-12-17 10:20:26 -08:00
Ryan Smith
c120b2633f Update README.md 2024-12-09 17:20:17 -08:00
48 changed files with 3741 additions and 872 deletions

2
.gitignore vendored
View File

@@ -27,4 +27,4 @@ deceptifeed.*
deceptifeed-*.*
# Ignore build output directory used by Makefile.
out/
bin/

View File

@@ -2,10 +2,11 @@
FROM golang:latest AS build-stage
WORKDIR /build
COPY . .
RUN git update-index -q --refresh
RUN make
FROM alpine:latest
RUN apk add --no-cache tzdata
WORKDIR /data
COPY --from=build-stage /build/out /
COPY --from=build-stage /build/bin /
ENTRYPOINT ["/deceptifeed"]

View File

@@ -1,22 +1,60 @@
# Makefile for Deceptifeed
TARGET_BINARY := ./out/deceptifeed
SOURCE := ./cmd/deceptifeed/
BIN_DIRECTORY := ./bin/
BIN_DEFAULT := $(BIN_DIRECTORY)deceptifeed
INSTALL_SCRIPT := ./scripts/install.sh
UNINSTALL_SCRIPT := ./scripts/install.sh --uninstall
VERSION := $(shell git describe --tags --dirty --broken)
BUILD_OPTIONS := -trimpath -ldflags="-s -w -X 'github.com/r-smith/deceptifeed/internal/config.Version=$(VERSION:v%=%)'"
GO := go
CGO_ENABLED := 0
GO111MODULE := on
.PHONY: build
build:
@echo "Building to: ./out/"
@mkdir -p ./out/
GO111MODULE=$(GO111MODULE) CGO_ENABLED=$(CGO_ENABLED) $(GO) build -o $(TARGET_BINARY) $(SOURCE)
@echo "Build complete."
@echo "Building for current operating system..."
@mkdir -p $(BIN_DIRECTORY)
CGO_ENABLED=$(CGO_ENABLED) $(GO) build $(BUILD_OPTIONS) -o $(BIN_DEFAULT) $(SOURCE)
@echo "Build complete: $(BIN_DEFAULT)"
@echo
.PHONY: all
all: build build-linux build-linux-arm build-freebsd build-windows
.PHONY: build-linux
build-linux:
@echo "Building for Linux..."
@mkdir -p $(BIN_DIRECTORY)
GOOS=linux GOARCH=amd64 CGO_ENABLED=$(CGO_ENABLED) $(GO) build $(BUILD_OPTIONS) -o $(BIN_DEFAULT)_linux_x64 $(SOURCE)
@echo "Build complete: $(BIN_DEFAULT)_linux_x64"
@echo
.PHONY: build-linux-arm
build-linux-arm:
@echo "Building for Linux (ARM)..."
@mkdir -p $(BIN_DIRECTORY)
GOOS=linux GOARCH=arm64 CGO_ENABLED=$(CGO_ENABLED) $(GO) build $(BUILD_OPTIONS) -o $(BIN_DEFAULT)_linux_ARM64 $(SOURCE)
@echo "Build complete: $(BIN_DEFAULT)_linux_ARM64"
@echo
.PHONY: build-freebsd
build-freebsd:
@echo "Building for FreeBSD..."
@mkdir -p $(BIN_DIRECTORY)
GOOS=freebsd GOARCH=amd64 CGO_ENABLED=$(CGO_ENABLED) $(GO) build $(BUILD_OPTIONS) -o $(BIN_DEFAULT)_freebsd_x64 $(SOURCE)
@echo "Build complete: $(BIN_DEFAULT)_freebsd_x64"
@echo
.PHONY: build-windows
build-windows:
@echo "Building for Windows..."
@mkdir -p $(BIN_DIRECTORY)
GOOS=windows GOARCH=amd64 CGO_ENABLED=$(CGO_ENABLED) $(GO) build $(BUILD_OPTIONS) -o $(BIN_DEFAULT)_windows_x64.exe $(SOURCE)
@echo "Build complete: $(BIN_DEFAULT)_windows_x64.exe"
@echo
.PHONY: install
install: $(TARGET_BINARY)
install: $(BIN_DEFAULT)
@bash $(INSTALL_SCRIPT)
.PHONY: uninstall
@@ -27,5 +65,5 @@ uninstall:
clean:
@echo "Cleaning started."
-@$(GO) clean
@rm --recursive --force ./out/
@rm --recursive --force $(BIN_DIRECTORY)
@echo "Cleaning complete."

150
README.md
View File

@@ -6,14 +6,16 @@
</picture>
</p>
`Deceptifeed` is a honeypot and threat feed server. It runs multiple honeypots (deceptive network services), while the threat feed lists IP addresses that have interacted with the honeypots.
`Deceptifeed` is a honeypot and threat feed server. It runs multiple deceptive network services (honeypots), while the threat feed lists IP addresses that have interacted with the honeypots. Additionally, `Deceptifeed` provides real-time visibility into honeypot activity, allowing you to monitor logs and interactions as they occur.
If an IP address interacts with a fake server on your network, why should it be allowed to access your real servers? `Deceptifeed` helps you build an automated defense system to reduce such risks. In a typical deployment, it runs alongside your real servers. The honeypots are exposed to the internet, while the threat feed remains private for use with your internal tools.
When an IP address interacts with a fake server on your network, why should it be allowed to access your real servers? `Deceptifeed` helps you build an automated defense system to reduce such risks. In a typical deployment, it runs alongside your real servers. The honeypots are exposed to the internet, while the threat feed remains private for use with your internal tools.
Most enterprise firewalls support ingesting threat feeds. By pointing to `Deceptifeed`, your firewall can automatically block IP addresses that interact with the honeypots. For other security tools, the threat feed is available in several formats, including plain text, CSV, JSON, and TAXII 2.1.
Most enterprise firewalls support ingesting threat feeds. By pointing to `Deceptifeed`, your firewall can automatically block IP addresses that interact with the honeypots. For other security tools, the threat feed is available in several formats, including plain text, CSV, JSON, and TAXII.
## Deployment Diagram
## Visuals
*Deployment diagram*
<a href="assets/diagram-light.svg?raw=true">
<picture>
@@ -23,9 +25,17 @@ Most enterprise firewalls support ingesting threat feeds. By pointing to `Decept
</picture>
</a>
<br>
<br>
<img alt="Example of the threat feed web interface" src="assets/screenshot-webfeed.png" width="860" />
<br>
<img alt="Example showing real-time honeypot log monitoring" src="assets/screenshot-live.png" width="860" />
## Quick Start
This section guides you through trying Deceptifeed as quickly as possible. There are no dependencies, configuration, or installation required. Refer to the [Installation](#installation) section when you're ready to set up a production environment.
This section guides you through trying Deceptifeed as quickly as possible. There are no dependencies, configuration, or installation required. Refer to the [Installation section](#installation) when you're ready to set up a production environment.
### Option 1: Download the binary
@@ -133,6 +143,7 @@ curl https://raw.githubusercontent.com/r-smith/deceptifeed/main/configs/docker-c
4. Run the Deceptifeed Docker container.
```shell
docker run --detach --name deceptifeed \
--env "TZ=America/Los_Angeles" \
--publish 2222:2222 \
--publish 8080:8080 \
--publish 8443:8443 \
@@ -144,36 +155,45 @@ deceptifeed/server:latest
Here is a breakdown of the arguments:
- `--detach` instructs Docker to run the Deceptifeed container in the background.
- `--env "TZ=xxx/yyy"` sets the time zone. Replace `xxx/yyy` with the TZ identifier for your local time zone. If you prefer UTC time, don't include this argument. Refer to this [list of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for valid TZ identifiers.
- `--publish ####:####` opens a network port on your host machine and maps it to Deceptifeed's Docker container. The first number specifies the port your host system listens on. You can set it to any open port. The second number specifies the port used by Deceptifeed inside the Docker container, which should match the ports configured in `config.xml`. There are multiple `--publish` arguments because Deceptifeed runs multiple network services. The default configuration includes an SSH honeypot on port 2222, an HTTP honeypot on port 8080, an HTTPS honeypot on port 8443, and the threat feed on port 9000. If you want your host machine to listen on port 443 for the HTTPS honeypot, for example, you would use the following line `--publish 443:8443 \`. This makes your host system listen on port 443 and maps it to the HTTPS honeypot defined for port 8443 in `config.xml`.
- `--restart unless-stopped` ensures Deceptifeed starts automatically when the host boots.
- `--volume /opt/deceptifeed/:/data/` specifies the directory on your host machine where persistent data is stored. If you used a different directory, adjust the path accordingly, but keep `:/data/` unchanged. For example: `--volume /path/to/deceptifeed/directory/:/data/ \`.
- `deceptifeed/server:latest` is the latest Docker image for Deceptifeed, hosted on *Docker Hub*. The image is updated with each official release and can be viewed on Docker Hub [here](https://hub.docker.com/r/deceptifeed/server).
- `deceptifeed/server:latest` is the latest Docker image for Deceptifeed, hosted on *Docker Hub*. The image is updated with each official release and can be viewed on [Docker Hub](https://hub.docker.com/r/deceptifeed/server).
## Features
- **Multiple Honeypot Servers:** Run any number of honeypot services simultaneously.
- **Threat Feed Server:** A real-time feed of IP addresses that have accessed your honeypots, delivered over HTTP. Available in plain text, CSV, JSON, STIX, and TAXII 2.1.
- **Rich Structured Logging:** Capture detailed logs of honeypot interactions in JSON format.
- **Secure:** The honeypot services never process or respond to client input; they only log the data received. Attackers are not given simulated or virtual environments.
- **Several Honeypot Types:**
- **SSH Honeyot:** Record login attempts to a fake SSH service.
- **HTTP/HTTPS Honeypot:** Record requested URLs and HTTP headers.
- **Generic TCP/UDP Services:** Record data sent by connecting clients.
- **Cross-platform:** Supports Linux, macOS, Windows, and *BSD.
- **Honeypot Servers:**
- Run any number of honeypot services simultaneously.
- Honeypots are low interaction (no simulated environments for attackers to access).
- **SSH honeypot:** Record and reject login attempts to a fake SSH service.
- **HTTP/HTTPS honeypot:** Record requested URLs and HTTP headers.
- **Generic TCP/UDP services:** Record data sent by connecting clients.
- **Threat Feed Server:**
- A feed of IP addresses that have accessed your honeypots, delivered over HTTP.
- Available in plain text, CSV, JSON, STIX, and TAXII.
- Includes a friendly web interface for browsing feed and honeypot data.
- **Rich Structured Logging:**
- Honeypot interactions are logged in a structured JSON format.
- Logs can be optionally forwarded to the SIEM of your choice.
- Automatic log file rollover ensures the system is self-managing.
- **Security:**
- The honeypot services never process or respond to client input.
- Attackers are not given simulated or virtual environments.
- Deceptifeed is self-contained and does **not** use any external libraries, frameworks, plugins, third-party modules, or GitHub actions.
- **Cross-platform:**
- Supports Linux, macOS, Windows, and *BSD.
- Available as a Docker container.
## Threat Feed
The threat feed provides a real-time list of IP addresses that have interacted with your honeypot services. It is delivered over HTTP for easy integration with firewalls. Most enterprise firewalls support ingesting custom threat feeds, allowing them to automatically block communication with the listed IP addresses.
The threat feed provides a list of IP addresses that have interacted with your honeypot services. It is delivered over HTTP for easy integration with firewalls. Most enterprise firewalls support ingesting custom threat feeds, allowing them to automatically block communication with the listed IP addresses.
Configure your firewall to use Deceptifeed as a custom threat feed and set your blocking rules accordingly. Ideally, exclude your honeypot services from any automatic blocking rules.
The threat feed is available in plain text, CSV, JSON, STIX, and TAXII 2.1.
**_Sample threat feed web interface_**
<img alt="Threat Feed Web Interface" src="assets/threatfeed-web-screenshot.png" width="881" />
The threat feed is available in plain text, CSV, JSON, STIX, and TAXII.
**_Sample threat feed in plain text_**
@@ -208,15 +228,15 @@ $ curl http://threatfeed.example.com:9000/json
"threat_feed": [
{
"ip": "10.32.16.110",
"added": "2024-11-12T16:18:36-08:00",
"last_seen": "2024-11-15T04:27:59-08:00",
"threat_score": 27
"added": "2025-02-12T16:18:36-08:00",
"last_seen": "2025-03-15T04:27:59-08:00",
"observations": 27
},
{
"ip": "192.168.2.21",
"added": "2024-11-14T23:09:11-08:00",
"last_seen": "2024-11-17T00:40:51-08:00",
"threat_score": 51
"added": "2025-04-02T23:09:11-08:00",
"last_seen": "2025-04-08T00:40:51-08:00",
"observations": 51
}
]
}
@@ -327,48 +347,72 @@ Due to the connectionless nature of UDP and the possibility of spoofed source in
## Upgrading
To upgrade Deceptifeed, follow the same steps you used for installation:
#### If you installed from the binary:
1. Download the latest package from the [Releases page](https://github.com/r-smith/deceptifeed/releases).
2. If you originally installed using the installation script, extract the latest package and re-run `install.sh`.
3. If you did not use the installation script, simply replace the existing `deceptifeed` binary with the new version.
#### If you installed from source:
### Binary
**If you originally installed using the installation script:**
1. Download the latest release from the [Releases page](https://github.com/r-smith/deceptifeed/releases).
2. Extract the files.
3. Run `install.sh`. The script will detect the existing installation and prompt you to upgrade.
- Optionally, you can add `--yes` to automatically confirm the upgrade prompt.
```shell
# Navigate to the directory where you cloned the `deceptifeed` repository:
cd #/path/to/deceptifeed/repository
# Extract:
tar xvzf <release>.tar.gz
# Update your local repository:
git pull origin main
# Change into the extracted directory:
cd deceptifeed
# Compile the code:
make
# Install (add `--yes` to auto-confirm the upgrade):
sudo ./install.sh
```
# Install the updated version:
sudo make install
**If you did not use the installation script:**
1. Download the latest release from the [Releases page](https://github.com/r-smith/deceptifeed/releases).
2. Extract the files.
3. Replace the existing `deceptifeed` binary with the new version.
### Docker
1. Pull the latest version of the Deceptifeed image:
```shell
docker pull deceptifeed/server:latest
```
2. Stop and remove the existing container:
```shell
docker stop deceptifeed
docker rm deceptifeed
```
3. Recreate the container with the new image:
```shell
# The `docker run` command will vary depending on how you originally ran the container.
# If you used the example from this documentation, it will look like this:
docker run --detach --name deceptifeed \
--env "TZ=America/Los_Angeles" \
--publish 2222:2222 \
--publish 8080:8080 \
--publish 8443:8443 \
--publish 9000:9000 \
--restart unless-stopped \
--volume /opt/deceptifeed/:/data/ \
deceptifeed/server:latest
```
## Uninstalling
#### If you installed from the binary:
### Binary
- If you used the installation script, re-run it with the `--uninstall` option.
**If you originally installed using the installation script:**
1. Re-run `install.sh` with the `--uninstall` option.
```shell
sudo ./install.sh --uninstall
```
- If you did not use the installation script, simply delete the `deceptifeed` binary and any generated files. When running the binary directly, any generated files will be named `deceptifeed-*` in the same directory where you ran the `deceptifeed` binary.
**If you did not use the installation script:**
1. Delete the `deceptifeed` binary and any generated files.
#### If you installed from source:
### Docker
```shell
# Navigate to the directory where you cloned the `deceptifeed` repository:
cd #/path/to/deceptifeed/repository
# Uninstall Deceptifeed:
sudo make uninstall
docker stop deceptifeed
docker rm deceptifeed
```

BIN
assets/screenshot-live.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 60 KiB

View File

@@ -1,10 +1,13 @@
package main
import (
"cmp"
"flag"
"fmt"
"log"
"os"
"slices"
"strconv"
"sync"
"github.com/r-smith/deceptifeed/internal/config"
@@ -40,8 +43,15 @@ func main() {
flag.StringVar(&https.CertPath, "https-cert", config.DefaultCertPathHTTPS, "Path to optional TLS public certificate")
flag.StringVar(&https.KeyPath, "https-key", config.DefaultKeyPathHTTPS, "Path to optional TLS private key")
flag.StringVar(&ssh.KeyPath, "ssh-key", config.DefaultKeyPathSSH, "Path to optional SSH private key")
ver := flag.Bool("version", false, "Output the version number and exit")
flag.Parse()
// If the `-version` flag is provided, output the version number and exit.
if *ver {
fmt.Println(config.Version)
return
}
// If the `-config` flag is not provided, use "config.xml" from the current
// directory if the file exists.
if len(*configPath) == 0 {
@@ -57,7 +67,7 @@ func main() {
if len(*configPath) > 0 {
cfgFromFile, err := config.Load(*configPath)
if err != nil {
log.Fatalln("Failed to load config:", err)
log.Fatalln("Shutting down. Failed to load configuration file:", err)
}
cfg = *cfgFromFile
} else {
@@ -65,16 +75,34 @@ func main() {
cfg.Servers = append(cfg.Servers, http, https, ssh)
// Set defaults.
for i := range cfg.Servers {
cfg.Servers[i].LogPath = cfg.LogPath
cfg.Servers[i].LogEnabled = true
cfg.Servers[i].SendToThreatFeed = true
cfg.Servers[i].ThreatScore = 1
if cfg.Servers[i].Type == config.SSH {
cfg.Servers[i].Banner = config.DefaultBannerSSH
}
}
}
// Sort the servers by port number. This is for cosmetic reasons to format
// the output when querying / viewing the active configuration.
slices.SortFunc(cfg.Servers, func(a, b config.Server) int {
p1, err := strconv.Atoi(a.Port)
if err != nil {
return 0
}
p2, err := strconv.Atoi(b.Port)
if err != nil {
return 0
}
t := cmp.Compare(p1, p2)
return t
})
// Initialize structured loggers for each honeypot server.
err := cfg.InitializeLoggers()
if err != nil {
log.Fatal("Shutting down. Error: ", err)
log.Fatalln("Shutting down. Failed to initialize logging:", err)
}
defer cfg.CloseLogFiles()
@@ -92,7 +120,7 @@ func main() {
return
}
threatfeed.Start(&cfg.ThreatFeed)
threatfeed.Start(&cfg)
}()
// Start the honeypot servers.

View File

@@ -12,10 +12,11 @@
<port>9000</port>
<databasePath>/opt/deceptifeed/logs/threatfeed.csv</databasePath>
<threatExpiryHours>336</threatExpiryHours>
<minimumThreatScore>0</minimumThreatScore>
<includePrivateIPs>false</includePrivateIPs>
<customThreatsPath></customThreatsPath>
<excludeListPath></excludeListPath>
<enableTLS>false</enableTLS>
<certPath>/opt/deceptifeed/certs/threatfeed-cert.pem</certPath>
<keyPath>/opt/deceptifeed/certs/threatfeed-key.pem</keyPath>
</threatFeed>
<!-- Honeypot Server Configuration -->
@@ -27,9 +28,8 @@
<port>2222</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<keyPath>/opt/deceptifeed/certs/ssh-key.pem</keyPath>
<banner>SSH-2.0-OpenSSH_9.3 FreeBSD-20230316</banner>
<banner>SSH-2.0-OpenSSH_9.6</banner>
</server>
<!-- HTTP honeypot server on port 8080 -->
@@ -38,7 +38,6 @@
<port>8080</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<rules>
<!-- Update the threat feed if any of the following rules match: -->
<include target="path" negate="true">(?i)^(/|/index\.html|/favicon\.ico|/robots\.txt|/sitemap\.xml|/\.well-known/\w+\.txt)$</include>
@@ -56,7 +55,6 @@
<port>8443</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<certPath>/opt/deceptifeed/certs/https-cert.pem</certPath>
<keyPath>/opt/deceptifeed/certs/https-key.pem</keyPath>
<rules>
@@ -77,7 +75,6 @@
<port>2323</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<banner>\nUser Access Verification\n\n</banner>
<prompts>
<prompt log="username">Username: </prompt>

View File

@@ -12,10 +12,11 @@
<port>9000</port>
<databasePath>threatfeed.csv</databasePath>
<threatExpiryHours>336</threatExpiryHours>
<minimumThreatScore>0</minimumThreatScore>
<includePrivateIPs>false</includePrivateIPs>
<customThreatsPath></customThreatsPath>
<excludeListPath></excludeListPath>
<enableTLS>false</enableTLS>
<certPath>key-threatfeed-public.pem</certPath>
<keyPath>key-threatfeed-private.pem</keyPath>
</threatFeed>
<!-- Honeypot Server Configuration -->
@@ -27,9 +28,8 @@
<port>2222</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<keyPath>key-ssh-private.pem</keyPath>
<banner>SSH-2.0-OpenSSH_9.3 FreeBSD-20230316</banner>
<banner>SSH-2.0-OpenSSH_9.6</banner>
</server>
<!-- HTTP honeypot server on port 8080 -->
@@ -38,7 +38,6 @@
<port>8080</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<rules>
<!-- Update the threat feed if any of the following rules match: -->
<include target="path" negate="true">(?i)^(/|/index\.html|/favicon\.ico|/robots\.txt|/sitemap\.xml|/\.well-known/\w+\.txt)$</include>
@@ -56,7 +55,6 @@
<port>8443</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<certPath>key-https-public.pem</certPath>
<keyPath>key-https-private.pem</keyPath>
<rules>
@@ -77,7 +75,6 @@
<port>2323</port>
<logEnabled>true</logEnabled>
<sendToThreatFeed>true</sendToThreatFeed>
<threatScore>1</threatScore>
<banner>\nUser Access Verification\n\n</banner>
<prompts>
<prompt log="username">Username: </prompt>

9
go.mod
View File

@@ -1,7 +1,10 @@
module github.com/r-smith/deceptifeed
go 1.22
go 1.24
require golang.org/x/crypto v0.30.0
require (
golang.org/x/crypto v0.38.0
golang.org/x/net v0.40.0
)
require golang.org/x/sys v0.28.0 // indirect
require golang.org/x/sys v0.33.0 // indirect

14
go.sum
View File

@@ -1,6 +1,8 @@
golang.org/x/crypto v0.30.0 h1:RwoQn3GkWiMkzlX562cLB7OxWvjH1L8xutO2WoJcRoY=
golang.org/x/crypto v0.30.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/crypto v0.38.0 h1:jt+WWG8IZlBnVbomuhg2Mdq0+BBQaHbtqHEFEigjUV8=
golang.org/x/crypto v0.38.0/go.mod h1:MvrbAqul58NNYPKnOra203SB9vpuZW0e+RRZV+Ggqjw=
golang.org/x/net v0.40.0 h1:79Xs7wF06Gbdcg4kdCCIQArK11Z1hr5POQ6+fIYHNuY=
golang.org/x/net v0.40.0/go.mod h1:y0hY0exeL2Pku80/zKK7tpntoX23cqL3Oa6njdgRtds=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.32.0 h1:DR4lr0TjUs3epypdhTOkMmuF5CDFJ/8pOnbzMZPQ7bg=
golang.org/x/term v0.32.0/go.mod h1:uZG1FhGx848Sqfsq4/DlJr3xGGsYMu/L5GW4abiaEPQ=

View File

@@ -0,0 +1,95 @@
package certutil
import (
"crypto/rand"
"crypto/rsa"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"fmt"
"math/big"
"os"
"time"
)
// GenerateSelfSigned creates a self-signed certificate and returns it as a
// tls.Certificate. If certPath and keyPath are provided, the generated
// certificate and private key are saved to disk.
func GenerateSelfSigned(certPath string, keyPath string) (tls.Certificate, error) {
// Generate 2048-bit RSA private key.
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to generate private key: %w", err)
}
// Set the certificate validity period to 10 years.
notBefore := time.Now()
notAfter := notBefore.AddDate(10, 0, 0)
// Generate a random certificate serial number.
serialNumber := make([]byte, 16)
_, err = rand.Read(serialNumber)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to generate certificate serial number: %w", err)
}
// Configure the certificate template.
template := x509.Certificate{
SerialNumber: new(big.Int).SetBytes(serialNumber),
Subject: pkix.Name{CommonName: "localhost"},
NotBefore: notBefore,
NotAfter: notAfter,
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
}
// Create the certificate.
derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &privateKey.PublicKey, privateKey)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to create certificate: %w", err)
}
certPEM := &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}
keyPEM := &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(privateKey)}
// Save the certificate and key to disk.
if len(certPath) > 0 && len(keyPath) > 0 {
// Silently ignore any potential errors and continue.
_ = writeCertAndKey(certPEM, keyPEM, certPath, keyPath)
}
return tls.X509KeyPair(pem.EncodeToMemory(certPEM), pem.EncodeToMemory(keyPEM))
}
// writeCertAndKey saves the public certificate and private key in PEM format
// to the specified paths.
func writeCertAndKey(cert *pem.Block, key *pem.Block, certPath string, keyPath string) error {
// Save the certificate file to disk.
certFile, err := os.Create(certPath)
if err != nil {
return err
}
defer certFile.Close()
if err := pem.Encode(certFile, cert); err != nil {
return err
}
// Save the private key file to disk.
keyFile, err := os.Create(keyPath)
if err != nil {
return err
}
defer keyFile.Close()
// Limit key access to the owner only.
_ = keyFile.Chmod(0600)
if err := pem.Encode(keyFile, key); err != nil {
return err
}
return nil
}

View File

@@ -6,9 +6,18 @@ import (
"io"
"log/slog"
"os"
"path/filepath"
"regexp"
"github.com/r-smith/deceptifeed/internal/logmonitor"
"github.com/r-smith/deceptifeed/internal/logrotate"
)
// Version stores Deceptifeed's version number. This variable is set at build
// time using the `-X` option with `-ldflags` and is assigned the latest Git
// tag. Refer to the Makefile in the project root for details on how it's set.
var Version = "undefined"
// This block of constants defines the default application settings when no
// configuration file is provided.
const (
@@ -28,7 +37,7 @@ const (
DefaultCertPathHTTPS = "deceptifeed-https.crt"
DefaultKeyPathHTTPS = "deceptifeed-https.key"
DefaultKeyPathSSH = "deceptifeed-ssh.key"
DefaultBannerSSH = "SSH-2.0-OpenSSH_9.3 FreeBSD-20230316" // SSH banner for FreeBSD 13.2
DefaultBannerSSH = "SSH-2.0-OpenSSH_9.6"
)
// ServerType represents the different types of honeypot servers that can be
@@ -75,31 +84,32 @@ func (t *ServerType) UnmarshalXMLAttr(attr xml.Attr) error {
// logger, settings for managing a threat feed, and the collection of honeypot
// servers that are configured to run.
type Config struct {
LogPath string `xml:"defaultLogPath"`
Servers []Server `xml:"honeypotServers>server"`
ThreatFeed ThreatFeed `xml:"threatFeed"`
LogPath string `xml:"defaultLogPath"`
Servers []Server `xml:"honeypotServers>server"`
ThreatFeed ThreatFeed `xml:"threatFeed"`
FilePath string `xml:"-"`
Monitor *logmonitor.Monitor `xml:"-"`
}
// Server represents a honeypot server with its relevant settings.
type Server struct {
Type ServerType `xml:"type,attr"`
Enabled bool `xml:"enabled"`
Port string `xml:"port"`
CertPath string `xml:"certPath"`
KeyPath string `xml:"keyPath"`
HomePagePath string `xml:"homePagePath"`
ErrorPagePath string `xml:"errorPagePath"`
Banner string `xml:"banner"`
Headers []string `xml:"headers>header"`
Prompts []Prompt `xml:"prompts>prompt"`
SendToThreatFeed bool `xml:"sendToThreatFeed"`
ThreatScore int `xml:"threatScore"`
Rules Rules `xml:"rules"`
SourceIPHeader string `xml:"sourceIpHeader"`
LogPath string `xml:"logPath"`
LogEnabled bool `xml:"logEnabled"`
LogFile *os.File
Logger *slog.Logger
Type ServerType `xml:"type,attr"`
Enabled bool `xml:"enabled"`
Port string `xml:"port"`
CertPath string `xml:"certPath"`
KeyPath string `xml:"keyPath"`
HomePagePath string `xml:"homePagePath"`
ErrorPagePath string `xml:"errorPagePath"`
Banner string `xml:"banner"`
Headers []string `xml:"headers>header"`
Prompts []Prompt `xml:"prompts>prompt"`
SendToThreatFeed bool `xml:"sendToThreatFeed"`
Rules Rules `xml:"rules"`
SourceIPHeader string `xml:"sourceIpHeader"`
LogPath string `xml:"logPath"`
LogEnabled bool `xml:"logEnabled"`
LogFile *logrotate.File `xml:"-"`
Logger *slog.Logger `xml:"-"`
}
type Rules struct {
@@ -129,14 +139,15 @@ type Prompt struct {
// can be configured to automatically block communication with IP addresses
// appearing in the threat feed.
type ThreatFeed struct {
Enabled bool `xml:"enabled"`
Port string `xml:"port"`
DatabasePath string `xml:"databasePath"`
ExpiryHours int `xml:"threatExpiryHours"`
IsPrivateIncluded bool `xml:"includePrivateIPs"`
MinimumThreatScore int `xml:"minimumThreatScore"`
CustomThreatsPath string `xml:"customThreatsPath"`
ExcludeListPath string `xml:"excludeListPath"`
Enabled bool `xml:"enabled"`
Port string `xml:"port"`
DatabasePath string `xml:"databasePath"`
ExpiryHours int `xml:"threatExpiryHours"`
IsPrivateIncluded bool `xml:"includePrivateIPs"`
ExcludeListPath string `xml:"excludeListPath"`
EnableTLS bool `xml:"enableTLS"`
CertPath string `xml:"certPath"`
KeyPath string `xml:"keyPath"`
}
// Load reads an optional XML configuration file and unmarshals its contents
@@ -151,22 +162,39 @@ func Load(filename string) (*Config, error) {
defer file.Close()
var config Config
absPath, err := filepath.Abs(filename)
if err != nil {
config.FilePath = filename
} else {
config.FilePath = absPath
}
xmlBytes, _ := io.ReadAll(file)
err = xml.Unmarshal(xmlBytes, &config)
if err != nil {
return nil, fmt.Errorf("failed to decode XML file: %w", err)
return nil, err
}
for i := range config.Servers {
// Ensure a minimum threat score of 0.
if config.Servers[i].ThreatScore < 0 {
config.Servers[i].ThreatScore = 0
// Use the global log path if the server log path is not specified.
if len(config.Servers[i].LogPath) == 0 {
config.Servers[i].LogPath = config.LogPath
}
// Validate regex rules.
if err := validateRegexRules(config.Servers[i].Rules); err != nil {
return nil, err
}
// Use the default SSH banner if no banner is specified.
if config.Servers[i].Type == SSH && len(config.Servers[i].Banner) == 0 {
config.Servers[i].Banner = DefaultBannerSSH
}
// Explicitly disable threat feed for UDP honeypots.
if config.Servers[i].Type == UDP {
config.Servers[i].SendToThreatFeed = false
}
}
return &config, nil
@@ -191,41 +219,59 @@ func validateRegexRules(rules Rules) error {
// files using the server's specified log path, defaulting to the global log
// path if none is provided.
func (c *Config) InitializeLoggers() error {
const maxSize = 50
c.Monitor = logmonitor.New()
openedLogFiles := make(map[string]*slog.Logger)
for i := range c.Servers {
if !c.Servers[i].Enabled {
continue
}
// Use the global log path if the server log path is not specified.
var logPath string
if len(c.Servers[i].LogPath) > 0 {
logPath = c.Servers[i].LogPath
} else {
logPath = c.LogPath
}
logPath := c.Servers[i].LogPath
// If no log path is given or if logging is disabled, set up a dummy
// logger to discard output.
// If no log path is specified or if logging is disabled, discard logs.
if len(logPath) == 0 || !c.Servers[i].LogEnabled {
c.Servers[i].Logger = slog.New(slog.NewTextHandler(io.Discard, nil))
c.Servers[i].Logger = slog.New(slog.DiscardHandler)
continue
}
// Open the specified log file and create a new logger.
logFile, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return fmt.Errorf("failed to open log file: %w", err)
// Check if this log path has already been opened. If so, reuse the
// logger.
if logger, exists := openedLogFiles[logPath]; exists {
c.Servers[i].Logger = logger
continue
}
c.Servers[i].LogFile = logFile
c.Servers[i].Logger = slog.New(slog.NewJSONHandler(c.Servers[i].LogFile, &slog.HandlerOptions{
ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
// Remove 'message' and 'log level' fields from output.
if a.Key == slog.MessageKey || a.Key == slog.LevelKey {
return slog.Attr{}
}
return a
},
}))
// Open the specified log file.
file, err := logrotate.OpenFile(logPath, maxSize)
if err != nil {
return err
}
// Create a JSON logger with two writers: one writes to disk using file
// rotation, the other writes to a channel for live monitoring.
logger := slog.New(
slog.NewJSONHandler(
io.MultiWriter(file, c.Monitor),
&slog.HandlerOptions{
ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
// Remove 'message' and 'log level' fields from output.
if a.Key == slog.MessageKey || a.Key == slog.LevelKey {
return slog.Attr{}
}
return a
},
},
),
)
c.Servers[i].Logger = logger
c.Servers[i].LogFile = file
// Store the logger for reuse.
openedLogFiles[logPath] = logger
}
return nil

29
internal/httpserver/fs.go Normal file
View File

@@ -0,0 +1,29 @@
package httpserver
import "io/fs"
// noDirectoryFS is a wrapper around fs.FS that disables directory listings.
type noDirectoryFS struct {
fs fs.FS
}
// Open opens the named file from the underlying fs.FS. The file is wrapped in
// a noReadDirFile to disable directory listings.
func (fs noDirectoryFS) Open(name string) (fs.File, error) {
f, err := fs.fs.Open(name)
if err != nil {
return nil, err
}
return noReadDirFile{f}, nil
}
// noReadDirFile wraps fs.File and overrides ReadDir to disable directory
// listings.
type noReadDirFile struct {
fs.File
}
// ReadDir always returns an error to disable directory listings.
func (noReadDirFile) ReadDir(int) ([]fs.DirEntry, error) {
return nil, fs.ErrInvalid
}

View File

@@ -2,46 +2,98 @@ package httpserver
import (
"context"
"crypto/rand"
"crypto/rsa"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"errors"
"fmt"
"io"
"io/fs"
"log"
"log/slog"
"math/big"
"net"
"net/http"
"net/netip"
"os"
"regexp"
"strings"
"time"
"github.com/r-smith/deceptifeed/internal/certutil"
"github.com/r-smith/deceptifeed/internal/config"
"github.com/r-smith/deceptifeed/internal/threatfeed"
)
// Start initializes and starts an HTTP or HTTPS honeypot server. The server
// is a simple HTTP server designed to log all details from incoming requests.
// Optionally, a single static HTML file can be served as the homepage,
// otherwise, the server will return only HTTP status codes to clients.
// Interactions with the HTTP server are sent to the threat feed.
// responseMode represents the HTTP response behavior for the honeypot.
// Depending on the configuration, the honeypot can serve a built-in default
// response, serve a specific file, or serve files from a specified directory.
type responseMode int
const (
modeDefault responseMode = iota // Serve the built-in default response.
modeFile // Serve a specific file.
modeDirectory // Serve files from a specified directory.
)
// responseConfig defines how the honeypot serves HTTP responses. It includes
// the response mode (default, file, or directory) and, for directory mode, an
// http.FileServer and file descriptor to the directory.
type responseConfig struct {
mode responseMode
fsRoot *os.Root
fsHandler http.Handler
}
// determineConfig reads the given configuration and returns a responseConfig,
// selecting the honeypot's response mode based on whether the HomePagePath
// setting is empty, a file, or a directory.
func determineConfig(cfg *config.Server) *responseConfig {
if len(cfg.HomePagePath) == 0 {
return &responseConfig{mode: modeDefault}
}
info, err := os.Stat(cfg.HomePagePath)
if err != nil {
return &responseConfig{mode: modeDefault}
}
if info.IsDir() {
root, err := os.OpenRoot(cfg.HomePagePath)
if err != nil {
return &responseConfig{mode: modeDefault}
}
return &responseConfig{
mode: modeDirectory,
fsRoot: root,
fsHandler: withCustomError(http.FileServerFS(noDirectoryFS{root.FS()}), cfg.ErrorPagePath),
}
}
return &responseConfig{
mode: modeFile,
}
}
// Start initializes and starts an HTTP or HTTPS honeypot server. It logs all
// request details and updates the threat feed as needed. If a filesystem path
// is specified in the configuration, the honeypot serves static content from
// the path.
func Start(cfg *config.Server) {
response := determineConfig(cfg)
if response.mode == modeDirectory {
defer response.fsRoot.Close()
}
switch cfg.Type {
case config.HTTP:
listenHTTP(cfg)
listenHTTP(cfg, response)
case config.HTTPS:
listenHTTPS(cfg)
listenHTTPS(cfg, response)
}
}
// listenHTTP initializes and starts an HTTP (plaintext) honeypot server.
func listenHTTP(cfg *config.Server) {
func listenHTTP(cfg *config.Server, response *responseConfig) {
mux := http.NewServeMux()
mux.HandleFunc("/", handleConnection(cfg, parseCustomHeaders(cfg.Headers)))
mux.HandleFunc("/", handleConnection(cfg, parseCustomHeaders(cfg.Headers), response))
srv := &http.Server{
Addr: ":" + cfg.Port,
Handler: mux,
@@ -58,10 +110,10 @@ func listenHTTP(cfg *config.Server) {
}
}
// listenHTTP initializes and starts an HTTPS (encrypted) honeypot server.
func listenHTTPS(cfg *config.Server) {
// listenHTTPS initializes and starts an HTTPS (encrypted) honeypot server.
func listenHTTPS(cfg *config.Server, response *responseConfig) {
mux := http.NewServeMux()
mux.HandleFunc("/", handleConnection(cfg, parseCustomHeaders(cfg.Headers)))
mux.HandleFunc("/", handleConnection(cfg, parseCustomHeaders(cfg.Headers), response))
srv := &http.Server{
Addr: ":" + cfg.Port,
Handler: mux,
@@ -72,10 +124,9 @@ func listenHTTPS(cfg *config.Server) {
}
// If the cert and key aren't found, generate a self-signed certificate.
if _, err := os.Stat(cfg.CertPath); os.IsNotExist(err) {
if _, err := os.Stat(cfg.KeyPath); os.IsNotExist(err) {
// Generate a self-signed certificate.
cert, err := generateSelfSignedCert(cfg.CertPath, cfg.KeyPath)
if _, err := os.Stat(cfg.CertPath); errors.Is(err, fs.ErrNotExist) {
if _, err := os.Stat(cfg.KeyPath); errors.Is(err, fs.ErrNotExist) {
cert, err := certutil.GenerateSelfSigned(cfg.CertPath, cfg.KeyPath)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to generate HTTPS certificate:", err)
return
@@ -93,104 +144,150 @@ func listenHTTPS(cfg *config.Server) {
}
}
// handleConnection is the handler for incoming HTTP and HTTPS client requests.
// It logs the details of each request and generates responses based on the
// requested URL. When the root or index.html is requested, it serves either an
// HTML file specified in the configuration or a default page prompting for
// basic HTTP authentication. Requests for any other URLs will return a 404
// error to the client.
func handleConnection(cfg *config.Server, customHeaders map[string]string) http.HandlerFunc {
// handleConnection processes incoming HTTP and HTTPS client requests. It logs
// the details of each request, updates the threat feed, and serves responses
// based on the honeypot configuration.
func handleConnection(cfg *config.Server, customHeaders map[string]string, response *responseConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// Log details of the incoming HTTP request.
// Log connection details. The log fields and format differ based on
// whether a custom source IP header is configured.
dst_ip, dst_port := getLocalAddr(r)
src_ip, _, _ := net.SplitHostPort(r.RemoteAddr)
username, password, isAuth := r.BasicAuth()
if isAuth {
cfg.Logger.LogAttrs(context.Background(), slog.LevelInfo, "",
logData := []slog.Attr{}
if len(cfg.SourceIPHeader) > 0 {
// A custom source IP header is configured. Set rem_ip to the
// original connecting IP and src_ip to the IP from the header. If
// the header is missing, invalid, contains multiple IPs, or if
// there a multiple headers with the same name, parsing will fail,
// and src_ip will fallback to the original connecting IP.
rem_ip := src_ip
header := r.Header[cfg.SourceIPHeader]
parsed := false
errMsg := ""
switch len(header) {
case 0:
errMsg = "missing header " + cfg.SourceIPHeader
case 1:
v := header[0]
if _, err := netip.ParseAddr(v); err != nil {
if strings.Contains(v, ",") {
errMsg = "multiple values in header " + cfg.SourceIPHeader
} else {
errMsg = "invalid IP in header " + cfg.SourceIPHeader
}
} else {
parsed = true
src_ip = v
}
default:
errMsg = "multiple instances of header " + cfg.SourceIPHeader
}
logData = append(logData,
slog.String("event_type", "http"),
slog.String("source_ip", src_ip),
slog.Bool("source_ip_parsed", parsed),
)
if !parsed {
logData = append(logData, slog.String("source_ip_error", errMsg))
}
logData = append(logData,
slog.String("remote_ip", rem_ip),
slog.String("server_ip", dst_ip),
slog.String("server_port", dst_port),
slog.String("server_name", config.GetHostname()),
slog.Group("event_details",
slog.String("method", r.Method),
slog.String("path", r.URL.Path),
slog.String("query", r.URL.RawQuery),
slog.String("user_agent", r.UserAgent()),
slog.String("protocol", r.Proto),
slog.String("host", r.Host),
slog.Group("basic_auth",
slog.String("username", username),
slog.String("password", password),
),
slog.Any("headers", flattenHeaders(r.Header)),
),
)
} else {
cfg.Logger.LogAttrs(context.Background(), slog.LevelInfo, "",
// No custom source IP header is configured. Log the standard
// connection details, keeping src_ip as the remote connecting IP.
logData = append(logData,
slog.String("event_type", "http"),
slog.String("source_ip", src_ip),
slog.String("server_ip", dst_ip),
slog.String("server_port", dst_port),
slog.String("server_name", config.GetHostname()),
slog.Group("event_details",
slog.String("method", r.Method),
slog.String("path", r.URL.Path),
slog.String("query", r.URL.RawQuery),
slog.String("user_agent", r.UserAgent()),
slog.String("protocol", r.Proto),
slog.String("host", r.Host),
slog.Any("headers", flattenHeaders(r.Header)),
)
}
// Log standard HTTP request information.
eventDetails := []any{
slog.String("method", r.Method),
slog.String("path", r.URL.Path),
slog.String("query", r.URL.RawQuery),
slog.String("user_agent", r.UserAgent()),
slog.String("protocol", r.Proto),
slog.String("host", r.Host),
slog.Any("headers", flattenHeaders(r.Header)),
}
// If the request includes a "basic" Authorization header, decode and
// log the credentials.
if username, password, ok := r.BasicAuth(); ok {
eventDetails = append(eventDetails,
slog.Group("basic_auth",
slog.String("username", username),
slog.String("password", password),
),
)
}
// Combine log data and write the final log entry.
logData = append(logData, slog.Group("event_details", eventDetails...))
cfg.Logger.LogAttrs(context.Background(), slog.LevelInfo, "", logData...)
// Print a simplified version of the request to the console.
fmt.Printf("[HTTP] %s %s %s %s\n", src_ip, r.Method, r.URL.Path, r.URL.RawQuery)
// Update the threat feed with the source IP address from the request.
// If the configuration specifies an HTTP header to be used for the
// source IP, retrieve the header value and use it instead of the
// connecting IP.
// Update the threat feed using the source IP address (src_ip). If a
// custom header is configured, src_ip contains the IP extracted from
// the header. Otherwise, it contains the remote connecting IP.
if shouldUpdateThreatFeed(cfg, r) {
src := src_ip
if len(cfg.SourceIPHeader) > 0 {
if header := r.Header.Get(cfg.SourceIPHeader); len(header) > 0 {
src = header
}
}
threatfeed.Update(src, cfg.ThreatScore)
threatfeed.Update(src_ip)
}
// Apply any custom HTTP response headers.
// Apply optional custom HTTP response headers.
for header, value := range customHeaders {
w.Header().Set(header, value)
}
// Serve a response based on the requested URL. If the root URL or
// /index.html is requested, serve the homepage. For all other
// requests, serve the error page with a 404 Not Found response.
// Optionally, a single static HTML file may be specified for both the
// homepage and the error page. If no custom files are provided,
// default minimal responses will be served.
if r.URL.Path == "/" || r.URL.Path == "/index.html" {
// Serve the homepage response.
if len(cfg.HomePagePath) > 0 {
http.ServeFile(w, r, cfg.HomePagePath)
} else {
// Serve a response based on the honeypot configuration.
switch response.mode {
case modeDefault:
// Built-in default response.
if r.URL.Path == "/" || r.URL.Path == "/index.html" {
if _, _, ok := r.BasicAuth(); ok {
time.Sleep(2 * time.Second)
}
w.Header()["WWW-Authenticate"] = []string{"Basic"}
w.WriteHeader(http.StatusUnauthorized)
} else {
serveErrorPage(w, r, cfg.ErrorPagePath)
}
} else {
// Serve the error page response.
w.WriteHeader(http.StatusNotFound)
if len(cfg.ErrorPagePath) > 0 {
http.ServeFile(w, r, cfg.ErrorPagePath)
case modeFile:
// Serve a single file.
if r.URL.Path == "/" || r.URL.Path == "/index.html" {
http.ServeFile(w, r, cfg.HomePagePath)
} else {
serveErrorPage(w, r, cfg.ErrorPagePath)
}
case modeDirectory:
// Serve files from a directory.
response.fsHandler.ServeHTTP(w, r)
}
}
}
// serveErrorPage serves an error HTTP response code and optional html page.
func serveErrorPage(w http.ResponseWriter, r *http.Request, path string) {
if len(path) == 0 {
w.WriteHeader(http.StatusNotFound)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
http.ServeFile(w, r, path)
}
// shouldUpdateThreatFeed determines if the threat feed should be updated based
// on the server's configured rules.
func shouldUpdateThreatFeed(cfg *config.Server, r *http.Request) bool {
@@ -280,7 +377,7 @@ func flattenHeaders(headers map[string][]string) map[string]string {
}
}
// Delete the User-Agent header, as it is managed separately.
delete(newHeaders, "User-Agent")
delete(newHeaders, "user-agent")
return newHeaders
}
@@ -295,91 +392,3 @@ func getLocalAddr(r *http.Request) (ip string, port string) {
}
return ip, port
}
// generateSelfSignedCert creates a self-signed TLS certificate and private key
// and returns the resulting tls.Certificate. If file paths are provided, the
// certificate and key are also saved to disk.
func generateSelfSignedCert(certPath string, keyPath string) (tls.Certificate, error) {
// Generate 2048-bit RSA private key.
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to generate private key: %w", err)
}
// Set the certificate validity period to 10 years.
notBefore := time.Now()
notAfter := notBefore.AddDate(10, 0, 0)
// Generate a random serial number for the certificate.
serialNumber := make([]byte, 16)
_, err = rand.Read(serialNumber)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to generate certificate serial number: %w", err)
}
// Set up the template for creating the certificate.
template := x509.Certificate{
SerialNumber: new(big.Int).SetBytes(serialNumber),
Subject: pkix.Name{CommonName: "localhost"},
NotBefore: notBefore,
NotAfter: notAfter,
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
BasicConstraintsValid: true,
}
// Use the template to create a self-signed X.509 certificate.
derBytes, err := x509.CreateCertificate(rand.Reader, &template, &template, &privateKey.PublicKey, privateKey)
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to create certificate: %w", err)
}
certPEM := &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}
keyPEM := &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(privateKey)}
// Save the certificate and key to disk.
if len(certPath) > 0 && len(keyPath) > 0 {
_ = writeCertAndKey(certPEM, keyPEM, certPath, keyPath)
// If saving fails, ignore the errors and use the in-memory
// certificate.
}
// Parse the public certificate and private key bytes into a tls.Certificate.
cert, err := tls.X509KeyPair(pem.EncodeToMemory(certPEM), pem.EncodeToMemory(keyPEM))
if err != nil {
return tls.Certificate{}, fmt.Errorf("failed to load certificate and private key: %w", err)
}
// Return the tls.Certificate.
return cert, nil
}
// writeCertAndKey saves the public certificate and private key in PEM format
// to the specified file paths.
func writeCertAndKey(cert *pem.Block, key *pem.Block, certPath string, keyPath string) error {
// Save the certificate file to disk.
certFile, err := os.Create(certPath)
if err != nil {
return err
}
defer certFile.Close()
if err := pem.Encode(certFile, cert); err != nil {
return err
}
// Save the private key file to disk.
keyFile, err := os.Create(keyPath)
if err != nil {
return err
}
defer keyFile.Close()
// Limit key access to the owner only.
_ = keyFile.Chmod(0600)
if err := pem.Encode(keyFile, key); err != nil {
return err
}
return nil
}

View File

@@ -0,0 +1,48 @@
package httpserver
import (
"net/http"
)
// withCustomError is a middleware that intercepts 4xx/5xx HTTP error responses
// and replaces them with a custom error response.
func withCustomError(next http.Handler, errorPath string) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
e := &errorInterceptor{origWriter: w, origRequest: r, errorPath: errorPath}
next.ServeHTTP(e, r)
})
}
// errorInterceptor intercepts HTTP responses to override error status codes
// and to serve a custom error response.
type errorInterceptor struct {
origWriter http.ResponseWriter
origRequest *http.Request
overridden bool
errorPath string
}
// WriteHeader intercepts error response codes (4xx or 5xx) to serve a custom
// error response.
func (e *errorInterceptor) WriteHeader(statusCode int) {
if statusCode >= 400 && statusCode <= 599 {
e.overridden = true
serveErrorPage(e.origWriter, e.origRequest, e.errorPath)
return
}
e.origWriter.WriteHeader(statusCode)
}
// Write writes the response body only if the response code was not overridden.
// Otherwise, the body is discarded.
func (e *errorInterceptor) Write(b []byte) (int, error) {
if !e.overridden {
return e.origWriter.Write(b)
}
return 0, nil
}
// Header returns the response headers from the original ResponseWriter.
func (e *errorInterceptor) Header() http.Header {
return e.origWriter.Header()
}

View File

@@ -0,0 +1,34 @@
package logmonitor
// Monitor is an io.Writer that sends bytes written to its Write method to an
// underlying byte channel. This allows other packages to receive the data from
// the channel. Writes are non-blocking. If there is no receiver, the data is
// silently discarded.
//
// Monitor does not implement io.Closer. Once initialized, it is meant to run
// for the duration of the program. If needed, manually close `Channel` when
// finished.
type Monitor struct {
Channel chan []byte
}
// New creates a new Monitor ready for I/O operations. The underlying `Channel`
// should have a receiver to capture and process the data.
func New() *Monitor {
channel := make(chan []byte, 2)
return &Monitor{
Channel: channel,
}
}
// Write sends the bytes from p to the underlying Monitor's channel. If there
// is no receiver for the channel, the data is silently discarded. Write always
// returns n = len(p) and err = nil.
func (m *Monitor) Write(p []byte) (n int, err error) {
select {
case m.Channel <- p:
return len(p), nil
default:
return len(p), nil
}
}

View File

@@ -0,0 +1,123 @@
package logrotate
import (
"fmt"
"os"
"sync"
)
// File is an io.WriteCloser that supports appending data to a file and file
// rotation.
//
// The file is automatically rotated once the file size exceeds the maximum
// size limit (specified in megabytes). `File` should be created using the
// `OpenFile` function.
type File struct {
name string
file *os.File
maxSize int64
size int64
mu sync.Mutex
}
// OpenFile opens the named file for appending. If successful, methods on the
// returned File can be used for I/O. When writing to the file, it will
// automatically rotate once the file size exceeds the maxSize (specified in
// megabytes).
func OpenFile(name string, maxSize int) (*File, error) {
if maxSize < 1 {
return nil, fmt.Errorf("maxSize must be greater than 0")
}
// Open the file for appending.
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return nil, err
}
// Get the current file size.
stat, err := file.Stat()
if err != nil {
_ = file.Close()
return nil, err
}
return &File{
file: file,
name: name,
maxSize: int64(maxSize) * 1024 * 1024, // Convert to megabytes
size: stat.Size(),
}, nil
}
// rotate checks if the file size exceeds the maximum allowed size. If so, it
// renames the current file by appending ".1" to its name and opens a new file
// with the original name. If a file with the ".1" suffix already exists, it is
// replaced.
func (f *File) rotate() error {
if f.size > f.maxSize {
// Retrieve the file information for the current file to capture its
// permissions. Any errors encountered are handled later and do not
// affect the rotation process.
info, statErr := f.file.Stat()
// Close the current file.
if err := f.file.Close(); err != nil {
return fmt.Errorf("can't close file: %w", err)
}
// Rename the file with a ".1" suffix.
if err := os.Rename(f.name, f.name+".1"); err != nil {
return fmt.Errorf("can't rename file: %w", err)
}
// Open a new file with the original name.
file, err := os.OpenFile(f.name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return fmt.Errorf("can't create new file: %w", err)
}
// Apply the original permissions to the new file. This is a
// best-effort operation that only runs if the previous os.Stat call
// was successful. Any errors from chmod are ignored.
if statErr == nil {
_ = file.Chmod(info.Mode().Perm())
}
// Reassign file and reset the size.
f.file = file
f.size = 0
}
return nil
}
// Write writes len(b) bytes from b to the File. If the File's size exceeds its
// maxSize, the file is renamed, a new file is opened with the orginal name,
// and the write is applied to the new file. Write returns the number of bytes
// written and an error, if any. Write returns a non-nil error when n != len(b).
func (f *File) Write(b []byte) (n int, err error) {
f.mu.Lock()
defer f.mu.Unlock()
// Rotate the log file if needed.
err = f.rotate()
if err != nil {
return 0, fmt.Errorf("log rotate: %w", err)
}
// Write the data and update the size.
n, err = f.file.Write(b)
f.size += int64(n)
return n, err
}
// Close closes the File, rendering it unusable for I/O. Close will return an
// error if it has already been called.
func (f *File) Close() error {
f.mu.Lock()
defer f.mu.Unlock()
err := f.file.Close()
f.file = nil
return err
}

View File

@@ -0,0 +1,178 @@
package proxyproto
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"net"
"net/netip"
"strings"
"time"
)
// v1Signature is the byte representation of "PROXY ", which is the start of a
// Proxy Protocol v1 header.
var v1Signature = []byte{
0x50, 0x52, 0x4F, 0x58, 0x59, 0x20,
}
// v2Signature is a 12-byte constant which is the start of a Proxy Protocol v2
// header.
var v2Signature = []byte{
0x0D, 0x0A, 0x0D, 0x0A,
0x00, 0x0D, 0x0A, 0x51,
0x55, 0x49, 0x54, 0x0A,
}
// serverTimeout defines the duration after which connected clients are
// automatically disconnected, set to 2 seconds.
const serverTimeout = 2 * time.Second
// ReadHeader reads and parses a Proxy Protocol v1 or v2 header from conn. It
// extracts and returns the client IP address from the header. It sets a
// 2-second deadline on conn. If parsing fails, it returns an error. Callers
// should reset the deadline after this function returns to extend the timeout.
func ReadHeader(conn net.Conn) (string, error) {
conn.SetDeadline(time.Now().Add(serverTimeout))
reader := bufio.NewReader(conn)
peek, err := reader.Peek(12)
if err != nil {
return "", errors.New("failed to read proxy header data")
}
var clientIP string
// Determine the Proxy Protocol version and parse accordingly.
if bytes.Equal(peek, v2Signature) {
// Proxy Protocol version 2.
clientIP, err = parseVersion2(reader)
if err != nil {
return "", fmt.Errorf("proxy protocol v2: %w", err)
}
} else if bytes.HasPrefix(peek, v1Signature) {
// Proxy Protocol version 1.
clientIP, err = parseVersion1(reader)
if err != nil {
return "", fmt.Errorf("proxy protocol v1: %w", err)
}
} else {
// Not a Proxy Protocol header.
return "", errors.New("invalid or missing proxy protocol header")
}
// Ensure the header data was provided by a private IP address.
host, _, _ := net.SplitHostPort(conn.RemoteAddr().String())
if ip, err := netip.ParseAddr(host); err != nil || (!ip.IsPrivate() && !ip.IsLoopback()) {
return "", errors.New("proxy connection must originate from a private IP address")
}
return clientIP, nil
}
// parseVersion1 reads and parses a Proxy Protocol vesion 1 text header and
// returns the extracted source IP address.
func parseVersion1(r *bufio.Reader) (string, error) {
// Proxy Protocol v1 ends with a CRLF (\r\n) and contains no more than 108
// bytes (including the CRLF). Read up to the newline. The presence of a
// carriage return before the newline is not validated.
buf := make([]byte, 0, 108)
for {
b, err := r.ReadByte()
if err != nil {
return "", fmt.Errorf("can't read header: %w", err)
}
buf = append(buf, b)
if b == '\n' {
break
}
if len(buf) == 108 {
return "", errors.New("invalid header")
}
}
// Split into space-delimited parts. When address information is provided,
// this should be exactly 6 parts. Other formats are not supported.
parts := strings.Fields(string(buf))
if len(parts) != 6 {
return "", errors.New("invalid or unsupported format")
}
// Read the protocol part and validate the address part. Protocols other
// than TCP4 and TCP6 are not supported by this implementation.
switch parts[1] {
case "TCP4":
// Parse and validate as an IPv4 address.
if ip, err := netip.ParseAddr(parts[2]); err != nil || !ip.Is4() || !ip.IsValid() {
return "", errors.New("invalid ipv4 source address")
}
case "TCP6":
// Parse and validate as an IPv6 address.
if ip, err := netip.ParseAddr(parts[2]); err != nil || !ip.Is6() || !ip.IsValid() {
return "", errors.New("invalid ipv6 source address")
}
default:
return "", errors.New("invalid or unsupported proxied protocol")
}
// Return the IP address part.
return parts[2], nil
}
// parseVersion2 reads and parses a Proxy Protocol vesion 2 binary header and
// returns the extracted source IP address.
func parseVersion2(r *bufio.Reader) (string, error) {
// Read the first 16 bytes into a buffer. The first 12 bytes is the Proxy
// Protocol v2 signature. Byte 13 is the protocol version and command. Byte
// 14 is the transport protocol and address family. Bytes 15-16 is the
// length of the address data.
header := make([]byte, 16)
if _, err := io.ReadFull(r, header); err != nil {
return "", fmt.Errorf("can't read header: %w", err)
}
// Byte 13 must be 0x21. The upper four bits represent the proxy protocol
// version, which must be 0x2. The lower four bits specify the command -
// 0x1 (PROXY) is the only supported command in this implementation.
if header[12] != 0x21 {
return "", errors.New("unsupported proxy command or version data")
}
// Read bytes 15-16, which specify the length (in bytes) of the address
// data in big-endian format. The address data includes source/destination
// IPs and ports. Read the specified number of bytes into a buffer. The
// length may indicate that additional bytes are part of the header beyond
// the address data. These are Type-Length-Value (TLV) vectors, which are
// read, but ignored by this implementation.
addresses := make([]byte, binary.BigEndian.Uint16(header[14:16]))
if _, err := io.ReadFull(r, addresses); err != nil {
return "", fmt.Errorf("can't read address information: %w", err)
}
// Byte 14 is the transport protocol and address family. Only TCP/UDP
// over IPv4 and IPv6 are supported in this implementation.
addrType := header[13]
// Extract, parse, validate, and return the source IP address.
// TCP over IPv4 = 0x11, UDP over IPv4 = 0x12.
if (addrType == 0x11 || addrType == 0x12) && len(addresses) >= 12 {
ip, ok := netip.AddrFromSlice(addresses[0:4])
if !ok || !ip.IsValid() {
return "", errors.New("invalid ipv4 source address")
}
return ip.String(), nil
}
// TCP over IPv6 = 0x21, UDP over IPv6 = 0x22.
if (addrType == 0x21 || addrType == 0x22) && len(addresses) >= 36 {
ip, ok := netip.AddrFromSlice(addresses[0:16])
if !ok || !ip.IsValid() {
return "", errors.New("invalid ipv6 source address")
}
return ip.String(), nil
}
return "", errors.New("unsupported transport protocol or address family")
}

View File

@@ -81,11 +81,11 @@ func Start(cfg *config.Server) {
)
// Print a simplified version of the request to the console.
fmt.Printf("[SSH] %s Username: %s Password: %s\n", src_ip, conn.User(), string(password))
fmt.Printf("[SSH] %s Username: %q Password: %q\n", src_ip, conn.User(), string(password))
// Update the threat feed with the source IP address from the request.
if cfg.SendToThreatFeed {
threatfeed.Update(src_ip, cfg.ThreatScore)
threatfeed.Update(src_ip)
}
// Insert fixed delay to mimic PAM.
@@ -139,7 +139,7 @@ func loadOrGeneratePrivateKey(path string) (ssh.Signer, error) {
// Load the specified file and return the parsed private key.
privateKey, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("failed to read private key from '%s': %w", path, err)
return nil, fmt.Errorf("failed to read private key '%s': %w", path, err)
}
signer, err := ssh.ParsePrivateKey(privateKey)
if err != nil {
@@ -150,20 +150,19 @@ func loadOrGeneratePrivateKey(path string) (ssh.Signer, error) {
// Generate and return a new private key.
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return nil, fmt.Errorf("failed to generate RSA private key: %w", err)
return nil, fmt.Errorf("failed to generate private key: %w", err)
}
// Save the private key to disk.
if len(path) > 0 {
// Silently ignore any potential errors and continue.
_ = writePrivateKey(path, privateKey)
// If saving fails, ignore the errors and use the in-memory private
// key.
}
// Convert the key to ssh.Signer.
signer, err := ssh.NewSignerFromKey(privateKey)
if err != nil {
return nil, fmt.Errorf("failed to convert RSA key to SSH signer: %w", err)
return nil, fmt.Errorf("failed to convert key to SSH signer: %w", err)
}
return signer, nil
} else {
@@ -171,8 +170,7 @@ func loadOrGeneratePrivateKey(path string) (ssh.Signer, error) {
}
}
// writePrivateKey saves a private key in PEM format to the specified file
// path.
// writePrivateKey saves a private key in PEM format to the specified path.
func writePrivateKey(path string, privateKey *rsa.PrivateKey) error {
privBytes := x509.MarshalPKCS1PrivateKey(privateKey)
privPem := &pem.Block{

View File

@@ -46,6 +46,7 @@ type Indicator struct {
ValidFrom time.Time `json:"valid_from"` // Required
ValidUntil *time.Time `json:"valid_until,omitempty"` // Optional
Name string `json:"name,omitempty"` // Optional
Confidence int `json:"confidence,omitempty"` // Optional
Description string `json:"description,omitempty"` // Optional
KillChains []KillChain `json:"kill_chain_phases,omitempty"` // Optional
Labels []string `json:"labels,omitempty"` // Optional
@@ -53,6 +54,24 @@ type Indicator struct {
CreatedByRef string `json:"created_by_ref,omitempty"` // Optional
}
// Sighting represents a STIX Sighting SRO.
type Sighting struct {
Type string `json:"type"` // Required
SpecVersion string `json:"spec_version"` // Required
ID string `json:"id"` // Required
Created time.Time `json:"created"` // Required
Modified time.Time `json:"modified"` // Required
FirstSeen time.Time `json:"first_seen"` // Optional
LastSeen time.Time `json:"last_seen"` // Optional
Count int `json:"count"` // Optional
Confidence int `json:"confidence,omitempty"` // Optional
Description string `json:"description,omitempty"` // Optional
Lang string `json:"lang,omitempty"` // Optional
SightingOfRef string `json:"sighting_of_ref"` // Required
WhereSightedRefs []string `json:"where_sighted_refs,omitempty"` // Optional
CreatedByRef string `json:"created_by_ref,omitempty"` // Optional
}
// KillChain represents a STIX `kill-chain-phase` type, which represents a
// phase in a kill chain.
type KillChain struct {

View File

@@ -4,7 +4,6 @@ import (
"crypto/rand"
"crypto/sha1"
"fmt"
prng "math/rand/v2"
)
var (
@@ -80,16 +79,10 @@ func newUUIDv4() string {
// overwritten to indicate version 4 and the variant (the format of the
// UUID).
// Get 16 random bytes.
// Get 16 random bytes. crypto/rand.Read is guaranteed to never return an
// error (as of Go 1.24).
var b = [16]byte{}
_, err := rand.Read(b[:])
if err != nil {
// Fall back to PRNG if the OS random number generator call fails.
for i := range b {
// Go's math/rand/v2 package is imported as `prng`.
b[i] = byte(prng.Int())
}
}
_, _ = rand.Read(b[:])
// Overwrite the version bits with 0b0100 (UUID version 4).
b[6] = (b[6] & 0x0f) | 0x40

View File

@@ -15,14 +15,20 @@ const (
IndicatorsID = "2cc72f88-8d92-4745-9c00-ea0deac18163"
// IndicatorsAlias is the friendly alias for the indicators collection.
IndicatorsAlias = "deceptifeed-indicators"
IndicatorsAlias = "indicators"
// ObservablesID is a fixed (random) identifier for the observables
// collection.
ObservablesID = "8aaff655-40de-41e2-9064-3dc1620d6420"
// ObservablesAlias is the friendly alias for the observables collection.
ObservablesAlias = "deceptifeed-observables"
ObservablesAlias = "observables"
// SightingsID is a fixed (random) identifier for the sightings collection.
SightingsID = "2b27973a-5891-4883-89aa-b7141e78e3e1"
// SightingsAlias is the friendly alias for the sightings collection.
SightingsAlias = "sightings"
)
// ImplementedCollections returns the collections that are available for use.
@@ -31,7 +37,7 @@ func ImplementedCollections() []Collection {
{
ID: IndicatorsID,
Title: "Deceptifeed Indicators",
Description: "This collection contains IP addresses represented as STIX Indicators",
Description: "This collection contains IP addresses observed interacting with honeypots, represented as STIX Indicators",
Alias: IndicatorsAlias,
CanRead: true,
CanWrite: false,
@@ -40,12 +46,21 @@ func ImplementedCollections() []Collection {
{
ID: ObservablesID,
Title: "Deceptifeed Observables",
Description: "This collection contains IP addresses represented as STIX Observables",
Description: "This collection contains IP addresses observed interacting with honeypots, represented as STIX Observables",
Alias: ObservablesAlias,
CanRead: true,
CanWrite: false,
MediaTypes: []string{ContentType},
},
{
ID: SightingsID,
Title: "Deceptifeed Sightings",
Description: "This collection contains Sightings of Indicators observed interacting with honeypots",
Alias: SightingsAlias,
CanRead: true,
CanWrite: false,
MediaTypes: []string{ContentType},
},
}
}

View File

@@ -112,11 +112,11 @@ func handleConnection(conn net.Conn, cfg *config.Server) {
)
// Print a simplified version of the interaction to the console.
fmt.Printf("[TCP] %s %v\n", src_ip, responsesToString(responses))
fmt.Printf("[TCP] %s %q\n", src_ip, responsesToString(responses))
// Update the threat feed with the source IP address from the interaction.
if cfg.SendToThreatFeed {
threatfeed.Update(src_ip, cfg.ThreatScore)
threatfeed.Update(src_ip)
}
}

View File

@@ -1,13 +1,14 @@
package threatfeed
import (
"bytes"
"bufio"
"encoding/csv"
"errors"
"math"
"net"
"fmt"
"net/netip"
"os"
"strconv"
"strings"
"sync"
"time"
)
@@ -22,15 +23,18 @@ type IOC struct {
// honeypot server.
lastSeen time.Time
// threatScore represents a score for a given IP address. It is incremented
// based on the configured threat score of the honeypot server that the IP
// interacted with.
threatScore int
// observations tracks the total number of interactions an IP has had with
// the honeypot servers.
observations int
}
const (
// dateFormat specifies the timestamp format used for threat feed entries.
dateFormat = time.RFC3339Nano
// maxObservations is the maximum number of interactions the threat feed
// will record for each IP.
maxObservations = 999_999_999
)
var (
@@ -42,63 +46,55 @@ var (
// based on the data in this map.
iocData = make(map[string]*IOC)
// mutex is to ensure thread-safe access to iocData.
mutex sync.Mutex
// mu is to ensure thread-safe access to iocData.
mu sync.Mutex
// dataChanged indicates whether the IoC map has been modified since the
// last time it was saved to disk.
dataChanged = false
// csvHeader defines the header row for saved threat feed data.
csvHeader = []string{"ip", "added", "last_seen", "threat_score"}
csvHeader = []string{"ip", "added", "last_seen", "observations"}
)
// Update updates the threat feed with the provided source IP address and
// threat score. This function should be called by honeypot servers whenever a
// client interacts with the honeypot. If the source IP address is already in
// the threat feed, its last-seen timestamp is updated, and its threat score is
// incremented. Otherwise, the IP address is added as a new entry in the threat
// feed.
func Update(ip string, threatScore int) {
// Update updates the threat feed with the provided source IP address. This
// function should be called by honeypot servers whenever a client interacts
// with the honeypot. If the source IP address is already in the threat feed,
// its last-seen timestamp is updated, and its observation count is
// incremented. Otherwise, the IP address is added as a new entry.
func Update(ip string) {
// Check if the given IP string is a private address. The threat feed may
// be configured to include or exclude private IPs.
netIP := net.ParseIP(ip)
if netIP == nil || netIP.IsLoopback() {
return
}
if !configuration.IsPrivateIncluded && netIP.IsPrivate() {
parsedIP, err := netip.ParseAddr(ip)
if err != nil || parsedIP.IsLoopback() || (!cfg.ThreatFeed.IsPrivateIncluded && parsedIP.IsPrivate()) {
return
}
now := time.Now()
mutex.Lock()
mu.Lock()
if ioc, exists := iocData[ip]; exists {
// Update existing entry.
ioc.lastSeen = now
if threatScore > 0 {
if ioc.threatScore > math.MaxInt-threatScore {
ioc.threatScore = math.MaxInt
} else {
ioc.threatScore += threatScore
}
if ioc.observations < maxObservations {
ioc.observations++
}
} else {
// Create a new entry.
iocData[ip] = &IOC{
added: now,
lastSeen: now,
threatScore: threatScore,
added: now,
lastSeen: now,
observations: 1,
}
}
mutex.Unlock()
mu.Unlock()
dataChanged = true
}
// deleteExpired deletes expired threat feed entries from the IoC map.
func deleteExpired() {
mutex.Lock()
defer mutex.Unlock()
mu.Lock()
defer mu.Unlock()
for key, value := range iocData {
if value.expired() {
@@ -110,26 +106,29 @@ func deleteExpired() {
// expired returns whether an IoC is considered expired based on the last
// seen date and the configured expiry hours.
func (ioc *IOC) expired() bool {
if configuration.ExpiryHours <= 0 {
if cfg.ThreatFeed.ExpiryHours <= 0 {
return false
}
return ioc.lastSeen.Before(time.Now().Add(-time.Hour * time.Duration(configuration.ExpiryHours)))
return ioc.lastSeen.Before(time.Now().Add(-time.Hour * time.Duration(cfg.ThreatFeed.ExpiryHours)))
}
// loadCSV loads existing threat feed data from a CSV file. If found, it
// populates iocData which represents the active threat feed. This function is
// called once during the initialization of the threat feed server.
func loadCSV() error {
file, err := os.Open(configuration.DatabasePath)
mu.Lock()
defer mu.Unlock()
f, err := os.Open(cfg.ThreatFeed.DatabasePath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return err
}
defer file.Close()
defer f.Close()
reader := csv.NewReader(file)
reader := csv.NewReader(f)
reader.FieldsPerRecord = -1
records, err := reader.ReadAll()
if err != nil {
@@ -141,7 +140,7 @@ func loadCSV() error {
var added time.Time
var lastSeen time.Time
var threatScore int
var count int
for _, record := range records[1:] {
ip := record[0]
@@ -157,17 +156,16 @@ func loadCSV() error {
lastSeen, _ = time.Parse(dateFormat, record[2])
}
// Parse threat score, defaulting to 1.
threatScore = 1
// Parse observation count, defaulting to 1.
count = 1
if len(record) > 3 && record[3] != "" {
if parsedLevel, err := strconv.Atoi(record[3]); err == nil {
threatScore = parsedLevel
if parsedCount, err := strconv.Atoi(record[3]); err == nil {
count = parsedCount
}
}
iocData[ip] = &IOC{added: added, lastSeen: lastSeen, threatScore: threatScore}
iocData[ip] = &IOC{added: added, lastSeen: lastSeen, observations: count}
}
deleteExpired()
return nil
}
@@ -175,29 +173,34 @@ func loadCSV() error {
// the threat feed data persists across application restarts. It is not the
// active threat feed.
func saveCSV() error {
buf := new(bytes.Buffer)
writer := csv.NewWriter(buf)
err := writer.Write(csvHeader)
f, err := os.OpenFile(cfg.ThreatFeed.DatabasePath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
if err != nil {
return err
}
defer f.Close()
w := bufio.NewWriterSize(f, 65536)
_, err = w.WriteString(strings.Join(csvHeader, ",") + "\n")
if err != nil {
return err
}
mutex.Lock()
mu.Lock()
for ip, ioc := range iocData {
if err := writer.Write([]string{
ip,
ioc.added.Format(dateFormat),
ioc.lastSeen.Format(dateFormat),
strconv.Itoa(ioc.threatScore),
}); err != nil {
_, err = w.WriteString(
fmt.Sprintf(
"%s,%s,%s,%d\n",
ip,
ioc.added.Format(dateFormat),
ioc.lastSeen.Format(dateFormat),
ioc.observations,
),
)
if err != nil {
return err
}
}
mutex.Unlock()
writer.Flush()
mu.Unlock()
if err := os.WriteFile(configuration.DatabasePath, buf.Bytes(), 0644); err != nil {
return err
}
return nil
return w.Flush()
}

View File

@@ -2,10 +2,9 @@ package threatfeed
import (
"bufio"
"bytes"
"cmp"
"fmt"
"net"
"net/netip"
"os"
"slices"
"strings"
@@ -16,11 +15,11 @@ import (
// feedEntry represents an individual entry in the threat feed.
type feedEntry struct {
IP string `json:"ip"`
IPBytes net.IP `json:"-"`
Added time.Time `json:"added"`
LastSeen time.Time `json:"last_seen"`
ThreatScore int `json:"threat_score"`
IP string `json:"ip"`
IPBytes netip.Addr `json:"-"`
Added time.Time `json:"added"`
LastSeen time.Time `json:"last_seen"`
Observations int `json:"observations"`
}
// feedEntries is a slice of feedEntry structs. It represents the threat feed
@@ -37,7 +36,7 @@ const (
byIP sortMethod = iota
byAdded
byLastSeen
byThreatScore
byObservations
)
// sortDirection represents the direction of sorting (ascending or descending).
@@ -72,27 +71,27 @@ func prepareFeed(options ...feedOptions) feedEntries {
opt = options[0]
}
excludedIPs, excludedCIDR, err := parseExcludeList(configuration.ExcludeListPath)
excludedIPs, excludedCIDR, err := parseExcludeList(cfg.ThreatFeed.ExcludeListPath)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to read threat feed exclude list:", err)
}
// Parse and filter IPs from iocData into the threat feed.
mutex.Lock()
mu.Lock()
threats := make(feedEntries, 0, len(iocData))
loop:
for ip, ioc := range iocData {
if ioc.expired() || ioc.threatScore < configuration.MinimumThreatScore || !ioc.lastSeen.After(opt.seenAfter) {
if ioc.expired() || !ioc.lastSeen.After(opt.seenAfter) {
continue
}
parsedIP := net.ParseIP(ip)
if parsedIP == nil || (parsedIP.IsPrivate() && !configuration.IsPrivateIncluded) {
parsedIP, err := netip.ParseAddr(ip)
if err != nil || (parsedIP.IsPrivate() && !cfg.ThreatFeed.IsPrivateIncluded) {
continue
}
for _, ipnet := range excludedCIDR {
if ipnet.Contains(parsedIP) {
for _, prefix := range excludedCIDR {
if prefix.Contains(parsedIP) {
continue loop
}
}
@@ -102,14 +101,14 @@ loop:
}
threats = append(threats, feedEntry{
IP: ip,
IPBytes: parsedIP,
Added: ioc.added,
LastSeen: ioc.lastSeen,
ThreatScore: ioc.threatScore,
IP: ip,
IPBytes: parsedIP,
Added: ioc.added,
LastSeen: ioc.lastSeen,
Observations: ioc.observations,
})
}
mutex.Unlock()
mu.Unlock()
threats.applySort(opt.sortMethod, opt.sortDirection)
@@ -118,31 +117,40 @@ loop:
// parseExcludeList reads IP addresses and CIDR ranges from a file. Each line
// should contain an IP address or CIDR. It returns a map of the unique IPs and
// a slice of the CIDR ranges found in the file.
func parseExcludeList(filepath string) (map[string]struct{}, []*net.IPNet, error) {
// a slice of the CIDR ranges found in the file. The file may include comments
// using "#". The "#" symbol on a line and everything after is ignored.
func parseExcludeList(filepath string) (map[string]struct{}, []netip.Prefix, error) {
if len(filepath) == 0 {
return map[string]struct{}{}, []*net.IPNet{}, nil
return nil, nil, nil
}
file, err := os.Open(filepath)
f, err := os.Open(filepath)
if err != nil {
return nil, nil, err
}
defer file.Close()
defer f.Close()
// `ips` stores individual IPs to exclude, and `cidr` stores CIDR networks
// to exclude.
ips := make(map[string]struct{})
cidr := []*net.IPNet{}
scanner := bufio.NewScanner(file)
cidr := []netip.Prefix{}
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if len(line) > 0 {
if _, ipnet, err := net.ParseCIDR(line); err == nil {
cidr = append(cidr, ipnet)
} else {
ips[line] = struct{}{}
}
line := scanner.Text()
// Remove comments and trim.
if i := strings.IndexByte(line, '#'); i != -1 {
line = line[:i]
}
line = strings.TrimSpace(line)
if len(line) == 0 {
continue
}
if prefix, err := netip.ParsePrefix(line); err == nil {
cidr = append(cidr, prefix)
} else {
ips[line] = struct{}{}
}
}
if err := scanner.Err(); err != nil {
@@ -157,13 +165,13 @@ func (f feedEntries) applySort(method sortMethod, direction sortDirection) {
switch method {
case byIP:
slices.SortFunc(f, func(a, b feedEntry) int {
return bytes.Compare(a.IPBytes, b.IPBytes)
return a.IPBytes.Compare(b.IPBytes)
})
case byLastSeen:
slices.SortFunc(f, func(a, b feedEntry) int {
t := a.LastSeen.Compare(b.LastSeen)
if t == 0 {
return bytes.Compare(a.IPBytes, b.IPBytes)
return a.IPBytes.Compare(b.IPBytes)
}
return t
})
@@ -171,15 +179,15 @@ func (f feedEntries) applySort(method sortMethod, direction sortDirection) {
slices.SortFunc(f, func(a, b feedEntry) int {
t := a.Added.Compare(b.Added)
if t == 0 {
return bytes.Compare(a.IPBytes, b.IPBytes)
return a.IPBytes.Compare(b.IPBytes)
}
return t
})
case byThreatScore:
case byObservations:
slices.SortFunc(f, func(a, b feedEntry) int {
t := cmp.Compare(a.ThreatScore, b.ThreatScore)
t := cmp.Compare(a.Observations, b.Observations)
if t == 0 {
return bytes.Compare(a.IPBytes, b.IPBytes)
return a.IPBytes.Compare(b.IPBytes)
}
return t
})
@@ -200,8 +208,8 @@ func (f feedEntries) convertToIndicators() []stix.Object {
result := make([]stix.Object, 0, len(f)+1)
// Add the Deceptifeed `Identity` as the first object in the collection.
// All IP addresses in the collection will reference this identity as
// the creator.
// All objects in the collection will reference this identity as the
// creator.
result = append(result, stix.DeceptifeedIdentity())
for _, entry := range f {
@@ -215,9 +223,9 @@ func (f feedEntries) convertToIndicators() []stix.Object {
validUntil := new(time.Time)
*validUntil = entry.LastSeen.AddDate(0, 2, 0).UTC()
// Generate a deterministic identifier for each IP address in the
// threat feed using the STIX IP pattern represented as a JSON
// string. For example: {"pattern":"[ipv4-addr:value='127.0.0.1']"}
// Generate a deterministic identifier using the IP address represented
// as a STIX IP pattern and structured as a JSON string. Example:
// {"pattern":"[ipv4-addr:value='127.0.0.1']"}
patternJSON := fmt.Sprintf("{\"pattern\":\"%s\"}", pattern)
result = append(result, stix.Indicator{
@@ -234,14 +242,67 @@ func (f feedEntries) convertToIndicators() []stix.Object {
Name: "Honeypot interaction: " + entry.IP,
Description: "This IP was observed interacting with a honeypot server.",
KillChains: []stix.KillChain{{KillChain: "mitre-attack", Phase: "reconnaissance"}},
Confidence: 100,
Lang: "en",
Labels: []string{"honeypot"},
Labels: []string{"honeypot-interaction"},
CreatedByRef: stix.DeceptifeedID,
})
}
return result
}
// convertToSightings converts IP addresses from the threat feed into a
// collection of STIX Sighting objects.
func (f feedEntries) convertToSightings() []stix.Object {
if len(f) == 0 {
return []stix.Object{}
}
const indicator = "indicator"
const sighting = "sighting"
const maxCount = 999_999_999 // Maximum count according to STIX 2.1 specification.
result := make([]stix.Object, 0, len(f)+1)
// Add the Deceptifeed `Identity` as the first object in the collection.
// All objects in the collection will reference this identity as the
// creator.
result = append(result, stix.DeceptifeedIdentity())
for _, entry := range f {
pattern := "[ipv4-addr:value = '"
if strings.Contains(entry.IP, ":") {
pattern = "[ipv6-addr:value = '"
}
pattern = pattern + entry.IP + "']"
count := min(entry.Observations, maxCount)
// Generate a deterministic identifier using the IP address represented
// as a STIX IP pattern and structured as a JSON string. Example:
// {"pattern":"[ipv4-addr:value='127.0.0.1']"}
indicatorJSON := fmt.Sprintf("{\"pattern\":\"%s\"}", pattern)
indicatorID := stix.DeterministicID(indicator, indicatorJSON)
result = append(result, stix.Sighting{
Type: sighting,
SpecVersion: stix.SpecVersion,
ID: stix.DeterministicID(sighting, "{\"sighting_of_ref\":\""+indicatorID+"\"}"),
Created: entry.Added.UTC(),
Modified: entry.LastSeen.UTC(),
FirstSeen: entry.Added.UTC(),
LastSeen: entry.LastSeen.UTC(),
Count: count,
Description: "This IP was observed interacting with a honeypot server.",
Confidence: 100,
Lang: "en",
SightingOfRef: indicatorID,
WhereSightedRefs: []string{stix.DeceptifeedID},
CreatedByRef: stix.DeceptifeedID,
})
}
return result
}
// convertToObservables converts IP addresses from the threat feed into a
// collection of STIX Cyber-observable Objects.
func (f feedEntries) convertToObservables() []stix.Object {
@@ -252,8 +313,8 @@ func (f feedEntries) convertToObservables() []stix.Object {
result := make([]stix.Object, 0, len(f)+1)
// Add the Deceptifeed `Identity` as the first object in the collection.
// All IP addresses in the collection will reference this identity as
// the creator.
// All objects in the collection will reference this identity as the
// creator.
result = append(result, stix.DeceptifeedIdentity())
for _, entry := range f {

View File

@@ -0,0 +1,485 @@
package threatfeed
import (
"cmp"
"encoding/json"
"io"
"net/http"
"os"
"slices"
"time"
)
// handleLogsMain serves a static page listing honeypot logs available for
// viewing.
func handleLogsMain(w http.ResponseWriter, r *http.Request) {
_ = parsedTemplates.ExecuteTemplate(w, "logs.html", "logs")
}
// handleLogs directs the request to the appropriate log parser based on the
// request path.
func handleLogs(w http.ResponseWriter, r *http.Request) {
switch r.PathValue("logtype") {
case "http":
switch r.PathValue("subtype") {
case "":
handleLogHTTP(w)
case "ip":
displayStats(w, httpIPStats{})
case "useragent":
displayStats(w, httpUserAgentStats{})
case "path":
displayStats(w, httpPathStats{})
case "query":
displayStats(w, httpQueryStats{})
case "method":
displayStats(w, httpMethodStats{})
case "host":
displayStats(w, httpHostStats{})
default:
handleNotFound(w, r)
}
case "ssh":
switch r.PathValue("subtype") {
case "":
handleLogSSH(w)
case "ip":
displayStats(w, sshIPStats{})
case "client":
displayStats(w, sshClientStats{})
case "username":
displayStats(w, sshUsernameStats{})
case "password":
displayStats(w, sshPasswordStats{})
default:
handleNotFound(w, r)
}
default:
handleNotFound(w, r)
}
}
// displayLogErrorPage servers an error page when there is a problem parsing
// log files.
func displayLogErrorPage(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusInternalServerError)
_ = parsedTemplates.ExecuteTemplate(w, "logs-error.html", map[string]any{"Error": err, "NavData": "logs"})
}
// handleLogSSH serves the SSH honeypot logs as a web page. It opens the
// honeypot log files, parses the data to JSON, and passes the result to an
// HTML template for rendering.
func handleLogSSH(w http.ResponseWriter) {
l := logFiles{}
reader, err := l.open()
if err != nil {
displayLogErrorPage(w, err)
return
}
defer l.close()
type Log struct {
Time time.Time `json:"time"`
EventType string `json:"event_type"`
SourceIP string `json:"source_ip"`
Details struct {
Username string `json:"username"`
Password string `json:"password"`
} `json:"event_details"`
}
const maxResults = 25_000
d := json.NewDecoder(reader)
data := make([]Log, 0, maxResults+1)
for d.More() {
var entry Log
err := d.Decode(&entry)
if err != nil || entry.EventType != "ssh" {
continue
}
data = append(data, entry)
if len(data) > maxResults {
data = data[1:]
}
}
slices.Reverse(data)
_ = parsedTemplates.ExecuteTemplate(w, "logs-ssh.html", map[string]any{"Data": data, "NavData": "logs"})
}
// handleLogHTTP serves the HTTP honeypot logs as a web page. It opens the
// honeypot log files, parses the data to JSON, and passes the result to an
// HTML template for rendering.
func handleLogHTTP(w http.ResponseWriter) {
l := logFiles{}
reader, err := l.open()
if err != nil {
displayLogErrorPage(w, err)
return
}
defer l.close()
type Log struct {
Time time.Time `json:"time"`
EventType string `json:"event_type"`
SourceIP string `json:"source_ip"`
Details struct {
Method string `json:"method"`
Path string `json:"path"`
} `json:"event_details"`
}
const maxResults = 25_000
d := json.NewDecoder(reader)
data := make([]Log, 0, maxResults+1)
for d.More() {
var entry Log
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
data = append(data, entry)
if len(data) > maxResults {
data = data[1:]
}
}
slices.Reverse(data)
_ = parsedTemplates.ExecuteTemplate(w, "logs-http.html", map[string]any{"Data": data, "NavData": "logs"})
}
// displayStats handles the processing and rendering of statistics for a given
// field. It reads honeypot log data, counts the occurrences of `field` and
// displays the results.
func displayStats(w http.ResponseWriter, field fieldCounter) {
l := logFiles{}
reader, err := l.open()
if err != nil {
displayLogErrorPage(w, err)
return
}
defer l.close()
fieldCounts := field.count(reader)
results := []statsResult{}
for k, v := range fieldCounts {
results = append(results, statsResult{Field: k, Count: v})
}
slices.SortFunc(results, func(a, b statsResult) int {
return cmp.Or(
-cmp.Compare(a.Count, b.Count),
cmp.Compare(a.Field, b.Field),
)
})
_ = parsedTemplates.ExecuteTemplate(
w,
"logs-stats.html",
map[string]any{
"Data": results,
"Header": field.fieldName(),
"NavData": "logs",
},
)
}
// statsResult holds a specific value for field and its associated count.
type statsResult struct {
Field string
Count int
}
// fieldCounter is an interface that defines methods for counting occurrences
// of specific fields.
type fieldCounter interface {
count(io.Reader) map[string]int
fieldName() string
}
// sshIPStats is the log structure for extracting SSH IP data.
type sshIPStats struct {
EventType string `json:"event_type"`
SourceIP string `json:"source_ip"`
}
func (sshIPStats) fieldName() string { return "Source IP" }
func (sshIPStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry sshIPStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "ssh" {
continue
}
fieldCounts[entry.SourceIP]++
}
return fieldCounts
}
// sshClientStats is the log structure for extracting SSH client data.
type sshClientStats struct {
EventType string `json:"event_type"`
Details struct {
Client string `json:"ssh_client"`
} `json:"event_details"`
}
func (sshClientStats) fieldName() string { return "SSH Client" }
func (sshClientStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry sshClientStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "ssh" {
continue
}
fieldCounts[entry.Details.Client]++
}
return fieldCounts
}
// sshUsernameStats is the log structure for extracting SSH username data.
type sshUsernameStats struct {
EventType string `json:"event_type"`
Details struct {
Username string `json:"username"`
} `json:"event_details"`
}
func (sshUsernameStats) fieldName() string { return "Username" }
func (sshUsernameStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry sshUsernameStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "ssh" {
continue
}
fieldCounts[entry.Details.Username]++
}
return fieldCounts
}
// sshPasswordStats is the log structure for extracting SSH password data.
type sshPasswordStats struct {
EventType string `json:"event_type"`
Details struct {
Password string `json:"password"`
} `json:"event_details"`
}
func (sshPasswordStats) fieldName() string { return "Password" }
func (sshPasswordStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry sshPasswordStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "ssh" {
continue
}
fieldCounts[entry.Details.Password]++
}
return fieldCounts
}
// httpIPStats is the log structure for extracting HTTP IP data.
type httpIPStats struct {
EventType string `json:"event_type"`
SourceIP string `json:"source_ip"`
}
func (httpIPStats) fieldName() string { return "Source IP" }
func (httpIPStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpIPStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.SourceIP]++
}
return fieldCounts
}
// httpUserAgentStats is the log structure for extracting HTTP user-agent data.
type httpUserAgentStats struct {
EventType string `json:"event_type"`
Details struct {
UserAgent string `json:"user_agent"`
} `json:"event_details"`
}
func (httpUserAgentStats) fieldName() string { return "User-Agent" }
func (httpUserAgentStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpUserAgentStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.Details.UserAgent]++
}
return fieldCounts
}
// httpPathStats is the log structure for extracting HTTP path data.
type httpPathStats struct {
EventType string `json:"event_type"`
Details struct {
Path string `json:"path"`
} `json:"event_details"`
}
func (httpPathStats) fieldName() string { return "Path" }
func (httpPathStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpPathStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.Details.Path]++
}
return fieldCounts
}
// httpQueryStats is the log structure for extracting HTTP query string data.
type httpQueryStats struct {
EventType string `json:"event_type"`
Details struct {
Query string `json:"query"`
} `json:"event_details"`
}
func (httpQueryStats) fieldName() string { return "Query String" }
func (httpQueryStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpQueryStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.Details.Query]++
}
return fieldCounts
}
// httpMethodStats is the log structure for extracting HTTP method data.
type httpMethodStats struct {
EventType string `json:"event_type"`
Details struct {
Method string `json:"method"`
} `json:"event_details"`
}
func (httpMethodStats) fieldName() string { return "HTTP Method" }
func (httpMethodStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpMethodStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.Details.Method]++
}
return fieldCounts
}
// httpHostStats is the log structure for extracting HTTP host header data.
type httpHostStats struct {
EventType string `json:"event_type"`
Details struct {
Host string `json:"host"`
} `json:"event_details"`
}
func (httpHostStats) fieldName() string { return "Host Header" }
func (httpHostStats) count(r io.Reader) map[string]int {
fieldCounts := map[string]int{}
d := json.NewDecoder(r)
for d.More() {
var entry httpHostStats
err := d.Decode(&entry)
if err != nil || entry.EventType != "http" {
continue
}
fieldCounts[entry.Details.Host]++
}
return fieldCounts
}
// logFiles represents open honeypot log files and their associate io.Reader.
type logFiles struct {
files []*os.File
readers []io.Reader
}
// open opens all honeypot log files and returns an io.MultiReader that
// combines all of the logs.
func (l *logFiles) open() (io.Reader, error) {
paths := []string{}
seenPaths := make(map[string]bool)
// Helper function to ensure only unique paths are added to the slice.
add := func(p string) {
if seenPaths[p] {
return
}
// New path. Add both the path and the path with ".1" to the slice.
paths = append(paths, p+".1", p)
seenPaths[p] = true
}
for _, s := range cfg.Servers {
add(s.LogPath)
}
for _, path := range paths {
f, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
continue
}
return nil, err
}
l.files = append(l.files, f)
}
for _, f := range l.files {
l.readers = append(l.readers, f)
}
return io.MultiReader(l.readers...), nil
}
// close closes all honeypot log files.
func (l *logFiles) close() {
for _, f := range l.files {
_ = f.Close()
}
}

View File

@@ -0,0 +1,95 @@
package threatfeed
import (
"fmt"
"net"
"net/http"
"net/netip"
"sync"
"golang.org/x/net/websocket"
)
// maxRecentMessages is the maximum number of recent log messages to store.
const maxRecentMessages = 100
var (
// muWSClients is to ensure threat-safe access to wsClients.
muWSClients sync.Mutex
// wsClients holds the connected WebSocket clients and is used to broadcast
// messages to all clients.
wsClients = make(map[*websocket.Conn]bool)
// wsRecentMessages stores the most recent log messages. These messages
// are sent to clients when they first connect.
wsRecentMessages = make([]string, 0, maxRecentMessages*1.5)
)
// handleLiveIndex serves a web page that displays honeypot log data in
// real-time through a WebSocket connection.
func handleLiveIndex(w http.ResponseWriter, r *http.Request) {
_ = parsedTemplates.ExecuteTemplate(w, "live.html", "live")
}
// broadcastLogsToClients receives honeypot log data through a byte channel
// configured to monitor the logs. When log data is received, the data is
// sent to all connected WebSocket clients. It also stores recent log data in a
// cache for newly connected clients.
func broadcastLogsToClients() {
for msg := range cfg.Monitor.Channel {
wsRecentMessages = append(wsRecentMessages, string(msg))
if len(wsRecentMessages) > maxRecentMessages {
wsRecentMessages = wsRecentMessages[1:]
}
muWSClients.Lock()
for client := range wsClients {
_ = websocket.Message.Send(client, string(msg))
}
muWSClients.Unlock()
}
}
// handleWebSocket establishes and maintains WebSocket connections with clients
// and performs cleanup when clients disconnect.
func handleWebSocket(ws *websocket.Conn) {
defer func() {
muWSClients.Lock()
delete(wsClients, ws)
muWSClients.Unlock()
_ = ws.Close()
}()
// Enforce private IPs.
ip, _, err := net.SplitHostPort(ws.Request().RemoteAddr)
if err != nil {
return
}
if parsedIP, err := netip.ParseAddr(ip); err != nil || (!parsedIP.IsPrivate() && !parsedIP.IsLoopback()) {
return
}
fmt.Println("[Threat Feed]", ip, "established WebSocket connection")
// Add newly connected client to map.
muWSClients.Lock()
wsClients[ws] = true
muWSClients.Unlock()
// Send the cache of recent log messages to the new client.
for _, msg := range wsRecentMessages {
_ = websocket.Message.Send(ws, msg)
}
// Send a message informing the client that we're done sending the initial
// cache of log messages.
_ = websocket.Message.Send(ws, "---end---")
// Keep WebSocket open.
var message string
for {
err := websocket.Message.Receive(ws, &message)
if err != nil {
break
}
}
}

View File

@@ -12,6 +12,7 @@ import (
"strings"
"time"
"github.com/r-smith/deceptifeed/internal/config"
"github.com/r-smith/deceptifeed/internal/stix"
"github.com/r-smith/deceptifeed/internal/taxii"
)
@@ -21,6 +22,11 @@ import (
//go:embed templates
var templates embed.FS
// parsedTemplates pre-parses and caches all HTML templates when the threat
// feed server starts. This eliminates the need for HTTP handlers to re-parse
// templates on each request.
var parsedTemplates = template.Must(template.ParseFS(templates, "templates/*.html"))
// handlePlain handles HTTP requests to serve the threat feed in plain text. It
// returns a list of IP addresses that interacted with the honeypot servers.
func handlePlain(w http.ResponseWriter, r *http.Request) {
@@ -38,21 +44,6 @@ func handlePlain(w http.ResponseWriter, r *http.Request) {
return
}
}
// If a custom threat file is supplied in the configuration, append the
// contents of the file to the HTTP response. To allow for flexibility, the
// contents of the file are not parsed or validated.
if len(configuration.CustomThreatsPath) > 0 {
data, err := os.ReadFile(configuration.CustomThreatsPath)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to read custom threats file:", err)
return
}
_, err = w.Write(data)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to serve threat feed:", err)
}
}
}
// handleJSON handles HTTP requests to serve the full threat feed in JSON
@@ -68,7 +59,7 @@ func handleJSON(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
e := json.NewEncoder(w)
e.SetIndent("", " ")
if err := e.Encode(map[string]interface{}{"threat_feed": prepareFeed(opt)}); err != nil {
if err := e.Encode(map[string]any{"threat_feed": prepareFeed(opt)}); err != nil {
fmt.Fprintln(os.Stderr, "Failed to encode threat feed to JSON:", err)
return
}
@@ -98,7 +89,7 @@ func handleCSV(w http.ResponseWriter, r *http.Request) {
entry.IP,
entry.Added.Format(dateFormat),
entry.LastSeen.Format(dateFormat),
strconv.Itoa(entry.ThreatScore),
strconv.Itoa(entry.Observations),
}); err != nil {
fmt.Fprintln(os.Stderr, "Failed to encode threat feed to CSV:", err)
return
@@ -111,11 +102,11 @@ func handleCSV(w http.ResponseWriter, r *http.Request) {
}
}
// handleSTIXIndicators handles HTTP requests to serve the full threat feed in
// STIX 2.1 format. The response includes all IoC data (IP addresses and their
// handleSTIX handles HTTP requests to serve the full threat feed in STIX 2.1
// format. The response includes all IoC data (IP addresses and their
// associated data). The response is structured as a STIX Bundle containing
// `Indicators` (STIX Domain Objects) for each IP address in the threat feed.
func handleSTIXIndicators(w http.ResponseWriter, r *http.Request) {
func handleSTIX(w http.ResponseWriter, r *http.Request) {
opt, err := parseParams(r)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
@@ -135,32 +126,6 @@ func handleSTIXIndicators(w http.ResponseWriter, r *http.Request) {
}
}
// handleSTIXObservables handles HTTP requests to serve a simplified version of
// the threat feed in STIX 2.1 format. The response is structured as a STIX
// Bundle containing `Observables` (STIX Cyber-observable Objects) for each IP
// address in the threat feed.
func handleSTIXObservables(w http.ResponseWriter, r *http.Request) {
opt, err := parseParams(r)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
const bundle = "bundle"
result := stix.Bundle{
Type: bundle,
ID: stix.NewID(bundle),
Objects: prepareFeed(opt).convertToObservables(),
}
w.Header().Set("Content-Type", stix.ContentType)
e := json.NewEncoder(w)
e.SetIndent("", " ")
if err := e.Encode(result); err != nil {
fmt.Fprintln(os.Stderr, "Failed to encode threat feed to STIX:", err)
}
}
// handleTAXIIDiscovery handles the TAXII server discovery endpoint, defined as
// `/taxii2/`. It returns a list of API root URLs available on the TAXII server.
// Deceptifeed has a single API root at `/taxii2/api/`
@@ -221,7 +186,7 @@ func handleTAXIICollections(w http.ResponseWriter, r *http.Request) {
return
}
} else {
result = map[string]interface{}{"collections": collections}
result = map[string]any{"collections": collections}
}
w.Header().Set("Content-Type", taxii.ContentType)
@@ -254,6 +219,8 @@ func handleTAXIIObjects(w http.ResponseWriter, r *http.Request) {
result.Objects = prepareFeed(opt).convertToIndicators()
case taxii.ObservablesID, taxii.ObservablesAlias:
result.Objects = prepareFeed(opt).convertToObservables()
case taxii.SightingsID, taxii.SightingsAlias:
result.Objects = prepareFeed(opt).convertToSightings()
default:
handleNotFound(w, r)
return
@@ -288,6 +255,8 @@ func handleTAXIIObjects(w http.ResponseWriter, r *http.Request) {
switch v := result.Objects[element].(type) {
case stix.Indicator:
timestamp = v.Modified
case stix.Sighting:
timestamp = v.LastSeen
case stix.ObservableIP:
if ioc, found := iocData[v.Value]; found {
timestamp = ioc.lastSeen
@@ -317,12 +286,35 @@ func handleTAXIIObjects(w http.ResponseWriter, r *http.Request) {
// delivers a static HTML document with information on accessing the threat
// feed.
func handleHome(w http.ResponseWriter, r *http.Request) {
tmpl := template.Must(template.ParseFS(templates, "templates/home.html"))
err := tmpl.Execute(w, nil)
_ = parsedTemplates.ExecuteTemplate(w, "home.html", "home")
}
// handleDocs serves a static page with documentation for accessing the threat
// feed.
func handleDocs(w http.ResponseWriter, r *http.Request) {
_ = parsedTemplates.ExecuteTemplate(w, "docs.html", "docs")
}
// handleCSS serves a CSS stylesheet for styling HTML templates.
func handleCSS(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/css")
data, err := templates.ReadFile("templates/css/style.css")
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to parse home page template:", err)
fmt.Println(err)
return
}
_, _ = w.Write(data)
}
// handleConfig serves a page that displays the Deceptifeed configuration.
func handleConfig(w http.ResponseWriter, r *http.Request) {
type templateData struct {
C config.Config
Version string
NavData string
}
d := templateData{C: cfg, Version: config.Version, NavData: "config"}
_ = parsedTemplates.ExecuteTemplate(w, "config.html", d)
}
// handleHTML returns the threat feed as a web page for viewing in a browser.
@@ -353,16 +345,15 @@ func handleHTML(w http.ResponseWriter, r *http.Request) {
m = "added"
case byLastSeen:
m = "last_seen"
case byThreatScore:
m = "threat_score"
case byObservations:
m = "observations"
}
tmpl := template.Must(template.ParseFS(templates, "templates/htmlfeed.html"))
err = tmpl.Execute(w, map[string]interface{}{"Data": prepareFeed(opt), "SortDirection": d, "SortMethod": m})
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to encode threat feed to HTML:", err)
return
}
_ = parsedTemplates.ExecuteTemplate(
w,
"webfeed.html",
map[string]any{"Data": prepareFeed(opt), "SortDirection": d, "SortMethod": m, "NavData": "webfeed"},
)
}
// paginate returns a slice of stix.Objects for the requested page, based on
@@ -435,8 +426,8 @@ func parseParams(r *http.Request) (feedOptions, error) {
opt.sortMethod = byLastSeen
case "added":
opt.sortMethod = byAdded
case "threat_score":
opt.sortMethod = byThreatScore
case "observations":
opt.sortMethod = byObservations
case "":
// No sort option specified.
default:
@@ -466,7 +457,8 @@ func parseParams(r *http.Request) (feedOptions, error) {
}
// handleNotFound returns a 404 Not Found response. This is the default
// response when a request is made outside the defined API.
// response when a request is made to an undefined path.
func handleNotFound(w http.ResponseWriter, r *http.Request) {
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
w.WriteHeader(http.StatusNotFound)
_ = parsedTemplates.ExecuteTemplate(w, "404.html", nil)
}

View File

@@ -3,6 +3,7 @@ package threatfeed
import (
"net"
"net/http"
"net/netip"
)
// enforcePrivateIP is a middleware that restricts access to the HTTP server
@@ -12,11 +13,11 @@ func enforcePrivateIP(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ip, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
http.Error(w, "Could not get IP", http.StatusInternalServerError)
http.Error(w, "", http.StatusForbidden)
return
}
if netIP := net.ParseIP(ip); !netIP.IsPrivate() && !netIP.IsLoopback() {
if parsedIP, err := netip.ParseAddr(ip); err != nil || (!parsedIP.IsPrivate() && !parsedIP.IsLoopback()) {
http.Error(w, "", http.StatusForbidden)
return
}

View File

@@ -1,12 +1,17 @@
package threatfeed
import (
"crypto/tls"
"errors"
"fmt"
"io/fs"
"net/http"
"os"
"time"
"github.com/r-smith/deceptifeed/internal/certutil"
"github.com/r-smith/deceptifeed/internal/config"
"golang.org/x/net/websocket"
)
const (
@@ -17,24 +22,23 @@ const (
)
var (
// configuration holds the configuration for the threat feed server. It is
// assigned when the server is initializing and the configuration values
// should not change.
configuration config.ThreatFeed
// cfg contains the application configuration. This includes settings for
// the threat feed server as well as for each individual honeypot server.
cfg config.Config
)
// Start initializes and starts the threat feed server. The server provides a
// list of IP addresses observed interacting with the honeypot servers in
// various formats.
func Start(cfg *config.ThreatFeed) {
configuration = *cfg
func Start(c *config.Config) {
cfg = *c
// Check for and open an existing threat feed CSV file, if available.
err := loadCSV()
if err != nil {
if err := loadCSV(); err != nil {
fmt.Fprintln(os.Stderr, "The Threat Feed server has stopped: Failed to open Threat Feed data:", err)
return
}
deleteExpired()
// Periodically delete expired entries and save the current threat feed to
// disk.
@@ -42,26 +46,33 @@ func Start(cfg *config.ThreatFeed) {
go func() {
for range ticker.C {
if dataChanged {
dataChanged = false
deleteExpired()
if err := saveCSV(); err != nil {
fmt.Fprintln(os.Stderr, "Error saving Threat Feed data:", err)
}
dataChanged = false
}
}
}()
// Monitor honeypot log data and broadcast to connected WebSocket clients.
go broadcastLogsToClients()
// Setup handlers and server configuration.
mux := http.NewServeMux()
mux.HandleFunc("GET /", enforcePrivateIP(handleNotFound))
mux.HandleFunc("GET /{$}", enforcePrivateIP(handleHome))
mux.HandleFunc("GET /feed", enforcePrivateIP(disableCache(handlePlain)))
mux.HandleFunc("GET /css/style.css", enforcePrivateIP(handleCSS))
mux.HandleFunc("GET /docs", enforcePrivateIP(handleDocs))
mux.HandleFunc("GET /config", enforcePrivateIP(handleConfig))
mux.HandleFunc("GET /live", enforcePrivateIP(handleLiveIndex))
mux.Handle("GET /live-ws", websocket.Handler(handleWebSocket))
// Threat feed handlers.
mux.HandleFunc("GET /webfeed", enforcePrivateIP(disableCache(handleHTML)))
mux.HandleFunc("GET /plain", enforcePrivateIP(disableCache(handlePlain)))
mux.HandleFunc("GET /csv", enforcePrivateIP(disableCache(handleCSV)))
mux.HandleFunc("GET /html", enforcePrivateIP(disableCache(handleHTML)))
mux.HandleFunc("GET /json", enforcePrivateIP(disableCache(handleJSON)))
mux.HandleFunc("GET /stix/indicators", enforcePrivateIP(disableCache(handleSTIXIndicators)))
mux.HandleFunc("GET /stix/observables", enforcePrivateIP(disableCache(handleSTIXObservables)))
mux.HandleFunc("GET /stix", enforcePrivateIP(disableCache(handleSTIX)))
// TAXII 2.1 handlers.
mux.HandleFunc("GET /taxii2/", enforcePrivateIP(handleNotFound))
mux.HandleFunc("POST /taxii2/", enforcePrivateIP(handleNotFound))
@@ -71,18 +82,45 @@ func Start(cfg *config.ThreatFeed) {
mux.HandleFunc("GET /taxii2/api/collections/{$}", enforcePrivateIP(handleTAXIICollections))
mux.HandleFunc("GET /taxii2/api/collections/{id}/{$}", enforcePrivateIP(handleTAXIICollections))
mux.HandleFunc("GET /taxii2/api/collections/{id}/objects/{$}", enforcePrivateIP(disableCache(handleTAXIIObjects)))
// Honeypot log handlers.
mux.HandleFunc("GET /logs", enforcePrivateIP(handleLogsMain))
mux.HandleFunc("GET /logs/{logtype}", enforcePrivateIP(handleLogs))
mux.HandleFunc("GET /logs/{logtype}/{subtype}", enforcePrivateIP(handleLogs))
srv := &http.Server{
Addr: ":" + cfg.Port,
Addr: ":" + c.ThreatFeed.Port,
Handler: mux,
ReadTimeout: 5 * time.Second,
WriteTimeout: 30 * time.Second,
IdleTimeout: 0,
}
// Start the threat feed HTTP server.
fmt.Printf("Starting Threat Feed server on port: %s\n", cfg.Port)
if err := srv.ListenAndServe(); err != nil {
fmt.Fprintln(os.Stderr, "The Threat Feed server has stopped:", err)
// Start the threat feed (HTTP) server if TLS is not enabled.
if !c.ThreatFeed.EnableTLS {
fmt.Printf("Starting threat feed (HTTP) on port: %s\n", c.ThreatFeed.Port)
if err := srv.ListenAndServe(); err != nil {
fmt.Fprintln(os.Stderr, "The threat feed server has stopped:", err)
}
return
}
// Generate a self-signed cert if the provided key and cert aren't found.
if _, err := os.Stat(c.ThreatFeed.CertPath); errors.Is(err, fs.ErrNotExist) {
if _, err := os.Stat(c.ThreatFeed.KeyPath); errors.Is(err, fs.ErrNotExist) {
cert, err := certutil.GenerateSelfSigned(c.ThreatFeed.CertPath, c.ThreatFeed.KeyPath)
if err != nil {
fmt.Fprintln(os.Stderr, "Failed to generate threat feed TLS certificate:", err)
return
}
// Add cert to server config.
srv.TLSConfig = &tls.Config{Certificates: []tls.Certificate{cert}}
}
}
// Start the threat feed (HTTPS) server.
fmt.Printf("Starting threat feed (HTTPS) on port: %s\n", c.ThreatFeed.Port)
if err := srv.ListenAndServeTLS(c.ThreatFeed.CertPath, c.ThreatFeed.KeyPath); err != nil {
fmt.Fprintln(os.Stderr, "The threat feed server has stopped:", err)
}
}

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,69 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body>
<header>
{{template "nav" .NavData}}
</header>
<main>
<article>
<h2>Configuration</h2>
<table class="config-info server-info">
<tbody>
<tr><th>Deceptifeed Version</th></tr>
<tr><td class="yellow">{{.Version}}</td></tr>
<tr><th>Configuration File</th></tr>
<tr><td>{{if .C.FilePath}}<span class="gray">{{.C.FilePath}}{{else}}<span class="red">(not set){{end}}</span></td></tr>
</tbody>
</table>
<table class="server-info">
<thead>
<tr><th class="cyan" colspan="2">Threat Feed</th></tr>
<tr><th class="gray" colspan="2">Port: <span class="orange">{{.C.ThreatFeed.Port}}</span></th></tr>
</thead>
<tbody>
<tr><th>State</th><td>{{if .C.ThreatFeed.Enabled}}<span class="green">Enabled{{else}}<span class="red">Disabled{{end}}</span></td></tr>
<tr><th>TLS</th><td>{{if .C.ThreatFeed.EnableTLS}}<span class="green">Enabled{{else}}<span class="red">Disabled{{end}}</span></td></tr>
{{if .C.ThreatFeed.EnableTLS}}<tr><th>Certificate</th><td class="blue">{{if .C.ThreatFeed.CertPath}}{{.C.ThreatFeed.CertPath}}{{else}}<span class="gray">(not set){{end}}</span></td></tr>{{end}}
{{if .C.ThreatFeed.EnableTLS}}<tr><th>Private Key</th><td class="blue">{{if .C.ThreatFeed.KeyPath}}{{.C.ThreatFeed.KeyPath}}{{else}}<span class="gray">(not set){{end}}</span></td></tr>{{end}}
<tr><th>Threat Database</th><td class="blue">{{if .C.ThreatFeed.DatabasePath}}{{.C.ThreatFeed.DatabasePath}}{{else}}<span class="gray">(not set)</span>{{end}}</td></tr>
<tr><th>Include Private IPs</th><td>{{if .C.ThreatFeed.IsPrivateIncluded}}<span class="red">Yes{{else}}<span class="green">No{{end}}</span></td></tr>
<tr><th>Expiry Hours</th><td class="orange">{{if eq .C.ThreatFeed.ExpiryHours 0}}<span class="gray">(never expire)</span>{{else}}{{.C.ThreatFeed.ExpiryHours}}{{end}}</td></tr>
<tr><th>Exclude List</th><td class="blue">{{if .C.ThreatFeed.ExcludeListPath}}{{.C.ThreatFeed.ExcludeListPath}}{{else}}<span class="gray">(not set)</span>{{end}}</td></tr>
</tbody>
</table>
{{range .C.Servers}}
<table class="server-info">
<thead>
<tr><th class="cyan" colspan="2"><span style="text-transform: uppercase;">{{.Type}}</span> Honeypot</th></tr>
<tr><th class="gray" colspan="2">Port: <span class="orange">{{.Port}}</span></th></tr>
</thead>
<tbody>
<tr><th>State</th><td>{{if .Enabled}}<span class="green">Enabled{{else}}<span class="red">Disabled{{end}}</span></td></tr>
<tr><th>Send to Threat Feed</th><td>{{if .SendToThreatFeed}}<span class="green">Enabled{{else}}<span class="red">Disabled{{end}}</span></td></tr>
<tr><th>Log State</th><td>{{if .LogEnabled}}<span class="green">Enabled{{else}}<span class="red">Disabled{{end}}</span></td></tr>
<tr><th>Log Path</th><td class="blue">{{if .LogPath}}{{.LogPath}}{{else}}<span class="gray">(not set)</span>{{end}}</td></tr>
{{if eq .Type.String "https"}}<tr><th>Certificate</th><td class="blue">{{if .CertPath}}{{.CertPath}}{{else}}<span class="gray">(not set){{end}}</span></td></tr>{{end}}
{{if or (eq .Type.String "https") (eq .Type.String "ssh")}}<tr><th>Private Key</th><td class="blue">{{if .KeyPath}}{{.KeyPath}}{{else}}<span class="gray">(not set){{end}}</span></td></tr>{{end}}
{{if .HomePagePath}}<tr><th>Home Page</th><td class="blue">{{.HomePagePath}}</td></tr>{{end}}
{{if .ErrorPagePath}}<tr><th>Error Page</th><td class="blue">{{.ErrorPagePath}}</td></tr>{{end}}
{{if .Banner}}<tr><th>Banner</th><td class="magenta">{{.Banner}}</td></tr>{{end}}
{{if .Headers}}<tr><th>Headers</th><td class="magenta">{{range .Headers}}{{.}}<br />{{end}}</td></tr>{{end}}
{{if .Prompts}}<tr><th>Prompts</th><td class="magenta">{{range .Prompts}}{{if .Text}}{{.Text}}<br />{{end}}{{end}}</td></tr>{{end}}
{{if .SourceIPHeader}}<tr><th>Source IP Header</th><td class="magenta">{{.SourceIPHeader}}</td></tr>{{end}}
{{if .Rules.Include}}{{range .Rules.Include}}<tr><th>Include Rule</th><td><span class="gray">Target:</span> <span class="white">{{.Target}}</span><br /><span class="gray">Negate:</span> <span class="white">{{.Negate}}</span><br /><span class="gray">Regex:</span> <span class="yellow">{{.Pattern}}</span></td></tr>{{end}}{{end}}
{{if .Rules.Exclude}}{{range .Rules.Exclude}}<tr><th>Exclude Rule</th><td><span class="gray">Target:</span> <span class="white">{{.Target}}</span><br /><span class="gray">Negate:</span> <span class="white">{{.Negate}}</span><br /><span class="gray">Regex:</span> <span class="yellow">{{.Pattern}}</span></td></tr>{{end}}{{end}}
</tbody>
</table>
{{end}}
</article>
</main>
</body>
</html>

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,129 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body>
<header>
{{template "nav" .}}
</header>
<main>
<article>
<h2>Threat Feed API</h2>
<table class="api-table">
<thead>
<tr>
<th>Endpoint</th>
<th>Format</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td row-header="Endpoint"><a class="endpoint" href="/plain">/plain</a></td>
<td row-header="Format"><span class="badge">Plain</span></td>
<td>One IP address per line. Suitable for firewall integration.</td>
</tr>
<tr>
<td row-header="Endpoint"><a class="endpoint" href="/csv">/csv</a></td>
<td row-header="Format"><span class="badge">CSV</span></td>
<td>CSV format containing full threat feed details.</td>
</tr>
<tr>
<td row-header="Endpoint"><a class="endpoint" href="/json">/json</a></td>
<td row-header="Format"><span class="badge">JSON</span></td>
<td>JSON format containing full threat feed details.</td>
</tr>
<tr>
<td row-header="Endpoint"><a class="endpoint" href="/stix">/stix</a></td>
<td row-header="Format"><span class="badge">STIX</span></td>
<td>STIX <em>Indicators</em> containing full threat feed details.</td>
</tr>
<tr>
<td row-header="Endpoint"><a class="endpoint" href="/taxii2">/taxii2</a></td>
<td row-header="Format"><span class="badge">TAXII</span></td>
<td>TAXII server. See the <em>TAXII</em> section for usage instructions.</td>
</tr>
</tbody>
</table>
<figure>
<figcaption><b>Example:</b> Retrieve the threat feed formatted as plain text:</figcaption>
<pre>curl "http://threatfeed.example.com:9000/plain"</pre>
</figure>
<figure>
<figcaption><b>Example:</b> Retrieve the threat feed formatted as JSON:</figcaption>
<pre>curl "http://threatfeed.example.com:9000/json"</pre>
</figure>
</article>
<article>
<h2>Query Parameters</h2>
<p>All endpoints support optional query parameters to customize how the threat feed is formatted.
The following query parameters are supported:</p>
<table class="docs-table">
<thead>
<tr>
<th>Parameter</th>
<th>Description</th>
</tr>
</thead>
<tbody class="align-top">
<tr>
<td><code>sort</code></td>
<td>Sort the results by a specific field. Valid values are:
<ul class="no-bullets">
<li><code>added</code></li>
<li><code>ip</code></li>
<li><code>last_seen</code></li>
<li><code>observations</code></li>
</ul>
</td>
</tr>
<tr>
<td><code>direction</code></td>
<td>Specify the sorting direction. Valid values are:
<ul class="no-bullets">
<li><code>asc</code> - Ascending order</li>
<li><code>desc</code> - Descending order</li>
</ul>
</td>
</tr>
<tr>
<td><code>last_seen_hours</code></td>
<td>Filter results to only include entries seen within the last specified number of hours.</td>
</tr>
</tbody>
</table>
<figure>
<figcaption><b>Example:</b> Retrieve the JSON feed, sorted by the last seen date in descending order:</figcaption>
<pre>curl "http://threatfeed.example.com:9000/json?sort=last_seen&direction=desc"</pre>
</figure>
<figure>
<figcaption><b>Example:</b> Retrieve the plain text feed, filtered to include only IP addresses seen within the last 24 hours:</figcaption>
<pre>curl "http://threatfeed.example.com:9000/plain?last_seen_hours=24"</pre>
</figure>
</article>
<article>
<h2>TAXII</h2>
<p>The threat feed is accessible via TAXII 2.1.
This allows for integration with Threat Intelligence Platforms (TIPs) like <i>OpenCTI</i> and <i>Microsoft Sentinel</i>.</p>
<p>To access via TAXII, clients require the API root URL and a collection ID.
The API root URL is available at the path <b>/taxii2/api/</b>.
Three collections are available: <b>indicators</b>, <b>sightings</b>, and <b>observables</b>.</p>
<h3>Key Details</h3>
<ul class="no-bullets">
<li>API root URL: <code>http://threatfeed.example.com:9000/taxii2/api/</code></li>
<li>Collection ID: <code>indicators</code></li>
<li>Collection ID: <code>observables</code></li>
<li>Collection ID: <code>sightings</code></li>
</ul>
</article>
</main>
</body>
</html>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,198 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body class="full-width">
<header>
{{template "nav" .}}
</header>
<main>
<div id="ws-status"></div>
<table id="logs" class="live-logs"></table>
</main>
<script>
const maxLogs = 200;
const logs = document.getElementById('logs');
const wsStatus = document.getElementById('ws-status');
const wsURL = '/live-ws';
const initialReconnectDelay = 1000;
const maxReconnectDelay = 15000;
const maxReconnectAttempts = 100;
const timeFormat = new Intl.DateTimeFormat([], {
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
});
let ws;
let reconnectAttempts = 0;
let isInitialBatchProcessed = false;
function handleWS() {
ws = new WebSocket(wsURL);
ws.onopen = () => {
reconnectAttempts = 0;
wsStatus.textContent = '';
wsStatus.className = '';
};
ws.onmessage = (event) => {
if (event.data === '---end---') {
isInitialBatchProcessed = true;
return;
}
handleMessage(event.data, isInitialBatchProcessed);
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
ws.onclose = () => {
reconnectWS();
};
}
function reconnectWS() {
if (reconnectAttempts > maxReconnectAttempts) {
wsStatus.textContent = 'Failed connecting to Deceptifeed';
wsStatus.className = 'red';
return;
}
const delay = Math.min(
maxReconnectDelay,
initialReconnectDelay * (2 ** reconnectAttempts),
);
const finalDelay = delay + (Math.random() * delay * 0.2);
setTimeout(() => {
reconnectAttempts++;
wsStatus.textContent = 'Connecting... ';
wsStatus.classList.add('gray', 'connecting');
handleWS();
}, finalDelay);
}
function handleMessage(data, shouldAnimate) {
try {
const d = JSON.parse(data);
const timeElement = document.createElement('td');
const initialTime = new Date(d.time);
timeElement.textContent = timeFormat.format(initialTime);
timeElement.className = 'timestamp';
const srcIPElement = document.createElement('td');
srcIPElement.textContent = d.source_ip;
srcIPElement.className = 'source-ip';
const eventDetails = document.createElement('td');
eventDetails.className = 'event-details';
switch (d.event_type) {
case 'http': {
const httpMethod = document.createElement('span');
httpMethod.textContent = `${d.event_details.method} `;
httpMethod.className = 'magenta';
const httpPath = document.createTextNode(d.event_details.path);
const tooltipContent = document.createElement('pre');
tooltipContent.className = 'tooltip-content';
let jsonDetails = JSON.stringify(d.event_details, null, 2);
// Remove outer braces.
jsonDetails = jsonDetails.slice(2, -1);
// Remove initial 2-space indent.
jsonDetails = jsonDetails.replace(/^ {2}/gm, '');
jsonDetails = jsonDetails.replace(/"([^"]+)":/g, '$1:');
tooltipContent.textContent = jsonDetails
eventDetails.classList.add('tooltip');
eventDetails.appendChild(httpMethod);
eventDetails.appendChild(httpPath);
eventDetails.appendChild(tooltipContent);
break;
}
case 'ssh': {
const usernameLabel = document.createElement('span');
usernameLabel.textContent = 'User: ';
usernameLabel.className = 'magenta';
const username = document.createTextNode(d.event_details.username);
const br = document.createElement('br');
const passwordLabel = document.createElement('span');
passwordLabel.textContent = 'Pass: ';
passwordLabel.className = 'magenta';
const password = document.createTextNode(d.event_details.password);
eventDetails.appendChild(usernameLabel);
eventDetails.appendChild(username);
eventDetails.appendChild(br);
eventDetails.appendChild(passwordLabel);
eventDetails.appendChild(password);
break;
}
case 'udp': {
// Remove '[unreliable]' string from IP found in UDP logs.
const spaceIndex = d.source_ip.indexOf(' ');
if (spaceIndex >= 0) {
srcIPElement.textContent = d.source_ip.slice(0, spaceIndex);
}
const udpLabel = document.createElement('span');
udpLabel.textContent = `[UDP:${d.server_port}] `;
udpLabel.className = 'magenta';
const udpData = document.createTextNode(d.event_details.data);
eventDetails.appendChild(udpLabel);
eventDetails.appendChild(udpData);
break;
}
default: {
const protoLabel = document.createElement('span');
protoLabel.textContent = `[${d.event_type.toUpperCase()}:${d.server_port}] `;
protoLabel.className = 'magenta';
const protoData = document.createTextNode(JSON.stringify(d.event_details, null, 1));
eventDetails.appendChild(protoLabel);
eventDetails.appendChild(protoData);
}
}
// Add log entry to table.
const logEntry = document.createElement('tr');
logEntry.appendChild(timeElement);
logEntry.appendChild(srcIPElement);
logEntry.appendChild(eventDetails);
if (shouldAnimate) {
logEntry.className = 'fade-in';
}
logs.insertBefore(logEntry, logs.firstChild);
if (logs.children.length > maxLogs) {
logs.removeChild(logs.lastChild);
}
}
catch (error) {
console.error('Failed to parse log data:', error);
}
}
handleWS();
</script>
</body>
</html>

View File

@@ -0,0 +1,22 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body>
<header>
{{template "nav" .NavData}}
</header>
<main>
<article>
<h1 class="error">Error opening log files</h1>
{{if .Error}}
<p class="error">{{.Error}}</p>
{{end}}
</article>
</main>
</body>
</html>

View File

@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body class="full-width">
<header>
{{template "nav" .NavData}}
</header>
<main class="full-width">
{{if .Data}}
<table class="logs logs-http">
<thead>
<tr><th>Time<th>Source IP<th>Method<th>Path</tr>
</thead>
<tbody>
{{range .Data}}<tr><td>{{.Time.Format "2006-01-02 15:04:05"}}<td>{{.SourceIP}}<td>{{.Details.Method}}<td>{{.Details.Path}}</tr>
{{end}}
</tbody>
</table>
{{else}}
<p class="no-results">No log data found</p>
{{end}}
</main>
</body>
</html>

View File

@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body class="full-width">
<header>
{{template "nav" .NavData}}
</header>
<main class="full-width">
{{if .Data}}
<table class="logs logs-ssh">
<thead>
<tr><th>Time<th>Source IP<th>Username<th>Password</tr>
</thead>
<tbody>
{{range .Data}}<tr><td class="time">{{.Time.Format "2006-01-02 15:04:05"}}<td>{{.SourceIP}}<td>{{.Details.Username}}<td>{{.Details.Password}}</tr>
{{end}}
</tbody>
</table>
{{else}}
<p class="no-results">No log data found</p>
{{end}}
</main>
</body>
</html>

View File

@@ -0,0 +1,47 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body class="full-width">
<header>
{{template "nav" .NavData}}
</header>
<main class="full-width">
{{if .Data}}
<table id="stats" class="logs logs-stats">
<thead>
<tr>
<th onclick="sortTable(0)">Count
<th onclick="sortTable(1)">{{.Header}}
</tr>
</thead>
<tbody>
{{range .Data}}<tr><td>{{.Count}}<td>{{.Field}}</tr>
{{end}}
</tbody>
</table>
{{else}}
<p class="no-results">No log data found</p>
{{end}}
</main>
<script>
function applyNumberSeparator() {
// Format 'Count' with a thousands separator based on user's locale.
const numberFormat = new Intl.NumberFormat();
document.querySelectorAll("#stats tbody tr").forEach(row => {
const observationCount = parseInt(row.cells[0].textContent, 10);
if (!isNaN(observationCount)) {
row.cells[0].textContent = numberFormat.format(observationCount);
}
});
}
applyNumberSeparator();
</script>
</body>
</html>

View File

@@ -0,0 +1,35 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body>
<header>
{{template "nav" .}}
</header>
<main>
<article>
<h2>Honeypot Logs</h2>
<ul class="log-list">
<li><a href="/logs/ssh">SSH Logs</a></li>
<li><a href="/logs/http">HTTP Logs</a></li>
</ul>
<ul class="log-list">
<li><a href="/logs/ssh/ip">Unique: SSH Source IPs</a></li>
<li><a href="/logs/ssh/username">Unique: SSH Usernames</a></li>
<li><a href="/logs/ssh/password">Unique: SSH Passwords</a></li>
<li><a href="/logs/ssh/client">Unique: SSH Clients</a></li>
<li><a href="/logs/http/ip">Unique: HTTP Source IPs</a></li>
<li><a href="/logs/http/useragent">Unique: HTTP User-Agents</a></li>
<li><a href="/logs/http/path">Unique: HTTP Paths</a></li>
<li><a href="/logs/http/query">Unique: HTTP Queries</a></li>
<li><a href="/logs/http/method">Unique: HTTP Methods</a></li>
<li><a href="/logs/http/host">Unique: HTTP Host Headers</a></li>
</ul>
</article>
</main>
</body>
</html>

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,93 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deceptifeed</title>
<link rel="stylesheet" href="/css/style.css">
</head>
<body class="full-width">
<header>
{{template "nav" .NavData}}
</header>
<main class="full-width">
{{if .Data}}
<table id="webfeed" class="webfeed">
<thead>
<tr>
<th><a href="?sort=ip&direction={{if and (eq .SortMethod "ip") (eq .SortDirection "asc")}}desc{{else}}asc{{end}}">
IP
</a>{{if eq .SortMethod "ip"}}<span class="sort-arrow {{if eq .SortDirection "asc"}}asc{{else}}desc{{end}}"></span>{{end}}
</th>
<th><a href="?sort=added&direction={{if and (eq .SortMethod "added") (eq .SortDirection "asc")}}desc{{else}}asc{{end}}">
Added
</a>{{if eq .SortMethod "added"}}<span class="sort-arrow {{if eq .SortDirection "asc"}}asc{{else}}desc{{end}}"></span>{{end}}
</th>
<th><a href="?sort=last_seen&direction={{if and (eq .SortMethod "last_seen") (eq .SortDirection "asc")}}desc{{else}}asc{{end}}">
Last Seen
</a>{{if eq .SortMethod "last_seen"}}<span class="sort-arrow {{if eq .SortDirection "asc"}}asc{{else}}desc{{end}}"></span>{{end}}
</th>
<th><a href="?sort=observations&direction={{if and (eq .SortMethod "observations") (eq .SortDirection "asc")}}desc{{else}}asc{{end}}">
Observations
</a>{{if eq .SortMethod "observations"}}<span class="sort-arrow {{if eq .SortDirection "asc"}}asc{{else}}desc{{end}}"></span>{{end}}
</th>
</tr>
</thead>
<tbody>
{{range .Data}}<tr><td>{{.IP}}<td>{{.Added.UTC.Format "2006-01-02T15:04:05.000Z"}}<td>{{.LastSeen.UTC.Format "2006-01-02T15:04:05.000Z"}}<td>{{.Observations}}
{{end}}
</tbody>
</table>
{{else}}
<p class="no-results">The threat feed is currently empty</p>
{{end}}
</main>
<script>
function formatDatesAndNumbers() {
// Format 'Added' as YYYY-MM-DD.
const addedDateFormat = new Intl.DateTimeFormat('en-CA', {
year: 'numeric',
month: '2-digit',
day: '2-digit',
});
// Format 'Last Seen' as YYYY-MM-DD hh:mm.
const lastSeenDateFormat = new Intl.DateTimeFormat('en-CA', {
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
hour12: false,
});
// Format 'Observations' with a thousands separator based on user's locale.
const numberFormat = new Intl.NumberFormat();
// Apply formats to table.
document.querySelectorAll("#webfeed tbody tr").forEach(row => {
// Apply format to 'Added' cell (row.cells[1]).
let date = new Date(row.cells[1].textContent);
if (!isNaN(date.valueOf())) {
row.cells[1].textContent = addedDateFormat.format(date);
}
// Apply format to 'Last Seen' cell (row.cells[2]).
date = new Date(row.cells[2].textContent);
if (!isNaN(date.valueOf())) {
row.cells[2].textContent = lastSeenDateFormat.format(date).replace(',', '');
}
// Apply format to 'Observations' cell (row.cells[3]).
const observationCount = parseInt(row.cells[3].textContent, 10);
if (!isNaN(observationCount)) {
row.cells[3].textContent = numberFormat.format(observationCount);
}
});
}
formatDatesAndNumbers();
</script>
</body>
</html>

View File

@@ -72,7 +72,7 @@ func Start(cfg *config.Server) {
)
// Print a simplified version of the interaction to the console.
fmt.Printf("[UDP] %s Data: %s\n", src_ip, strings.TrimSpace(string(buffer[:n])))
fmt.Printf("[UDP] %s Data: %q\n", src_ip, strings.TrimSpace(string(buffer[:n])))
}()
}
}

View File

@@ -158,9 +158,9 @@ install_app() {
if [[ -f "${script_dir}/${source_bin}" ]]; then
# Found in the same directory as the script.
source_bin="${script_dir}/${source_bin}"
elif [[ -f "${script_dir}/../out/${source_bin}" ]]; then
# Found in ../out relative to the script.
source_bin="${script_dir}/../out/${source_bin}"
elif [[ -f "${script_dir}/../bin/${source_bin}" ]]; then
# Found in ../bin relative to the script.
source_bin="${script_dir}/../bin/${source_bin}"
else
# Could not locate.
echo -e "${msg_error} ${white}Unable to locate the file: ${yellow}'${source_bin}'${clear}\n" >&2