Information Security Reference

L-SM-TH

Logging(Host/Network) / Security Monitoring / Threat Hunting

Table of Contents

101

Logging

Monitoring

  • 101
    • Articles/Blogposts/Writeups
    • Talks/Presentations/Videos
  • Breach Detection/Response
    • Articles/Blogposts/Presentations/Talks/Writeups
    • Tools
      • Infection Monkey
        • The Infection Monkey is an open source security tool for testing a data center's resiliency to perimeter breaches and internal server infection. The Monkey uses various methods to self propagate across a data center and reports success to a centralized Command and Control(C&C) server.
      • 411
        • Configure Searches to periodically run against a variety of data sources. You can define a custom pipeline of Filters to manipulate any generated Alerts and forward them to multiple Targets.
      • Pattern
        • Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google,; Twitter, Wikipedia), web crawler, HTML DOM parser; Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet; Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron); Network Analysis: graph centrality and visualization.
  • Infrastructure Monitoring
  • Network-based
    • 101
    • Articles/Writeups
    • Talks/Presentations
    • Understanding
    • Sigma
      • Sigma
        • Sigma is a generic and open signature format that allows you to describe relevant log events in a straight forward manner. The rule format is very flexible, easy to write and applicable to any type of log file. The main purpose of this project is to provide a structured form in which researchers or analysts can describe their once developed detection methods and make them shareable with others. Sigma is for log files what Snort is for network traffic and YARA is for files.
      • Sigma Specification
      • How to Write Sigma Rules - Florian Roth
      • Sigma - Generic Signatures for Log Events - Thomas Patzke(Hack.lu2017)
        • Log files are a great resource for hunting threats and analysis of incidents. Unfortunately, there is no standardized signature format like YARA for files or Snort signatures for network traffic. This makes sharing of log signatures by security researchers and software developers problematic. Further, most SIEM systems have their own query language, which makes signature distribution in large heterogeneous environments inefficient and increases costs for replacement of SIEM solutions.Sigma tries to fill these gaps by providing a YAML-based format for log signatures, an open repository of signatures and an extensible tool that converts Sigma signatures into different query languages. Rules and tools were released as open source and are actively developed. This presentation gives an overview about use cases, Sigma rules and the conversion tool, the development community and future plans of the project.
      • MITRE ATT&CK and Sigma Alerting - Justin Henderson, John Hubbard(2019)
        • This webcast will introduce the Sigma Alert project and show examples of creating alert rules against MITRE ATT&CK framework items to discover attacks in a way that works for multiple products. Sigma allows for writing rules in a neutral rule format that supports converting the rule to support your product of choice.
    • IDS/IPS Tools
      • Snort
      • Bro/Zeek
        • 101
        • Articles/Blogposts
        • Tools
          • bro-intel-generator
            • Script for generating Bro intel files from pdf or html reports
          • bro-domain-generation
            • Detect domain generation algorithms (DGA) with Bro. The module will regularly generate domains by any implemented algorithms and watch for those domains in DNS queries. This script only works with Bro 2.1+.
          • Exfil Framework
            • The Exfil Framework is a suite of Bro scripts that detect file uploads in TCP connections. The Exfil Framework can detect file uploads in most TCP sessions including sessions that have encrypted payloads (SCP,SFTP,HTTPS).
          • brim
            • Desktop application to efficiently search large packet captures and Zeek logs.
      • Suricata
      • Snort
      • Argus
      • Other
        • Maltrail
          • Maltrail is a malicious traffic detection system, utilizing publicly available (black)lists containing malicious and/or generally suspicious trails, along with static trails compiled from various AV reports and custom user defined lists, where trail can be anything from domain name (e.g. zvpprsensinaix.com for Banjori malware), URL (e.g. http://109.162.38.120/harsh02.exe for known malicious executable), IP address (e.g. 185.130.5.231 for known attacker) or HTTP User-Agent header value (e.g. sqlmap for automatic SQL injection and database takeover tool). Also, it uses (optional) advanced heuristic mechanisms that can help in discovery of unknown threats (e.g. new malware).
    • IDS/IPS Monitoring Tools
      • Snorby
      • Snorby - Github
        • Snorby is a ruby on rails web application for network security monitoring that interfaces with current popular intrusion detection systems (Snort, Suricata and Sagan). The basic fundamental concepts behind Snorby are simplicity, organization and power. The project goal is to create a free, open source and highly competitive application for network monitoring for both private and enterprise use.
      • Squil
        • Sguil (pronounced sgweel) is built by network security analysts for network security analysts. Sguil's main component is an intuitive GUI that provides access to realtime events, session data, and raw packet captures. Sguil facilitates the practice of Network Security Monitoring and event driven analysis. The Sguil client is written in tcl/tk and can be run on any operating system that supports tcl/tk (including Linux, * BSD, Solaris, MacOS, and Win32).
        • Squil FAQ
      • Squert
        • Squert is a web application that is used to query and view event data stored in a Sguil database (typically IDS alert data). Squert is a visual tool that attempts to provide additional context to events through the use of metadata, time series representations and weighted and logically grouped result sets. The hope is that these views will prompt questions that otherwise may not have been asked.
        • Slide Deck on Squert
        • Install/setup/etc - Github
    • General Tools
      • General
        • Security Onion
          • Security Onion is a Linux distro for intrusion detection, network security monitoring, and log management. It's based on Ubuntu and contains Snort, Suricata, Bro, OSSEC, Sguil, Squert, Snorby, ELSA, Xplico, NetworkMiner, and many other security tools. The easy-to-use Setup wizard allows you to build an army of distributed sensors for your enterprise in minutes!
      • Data Tranformation
        • Pip3line, the Swiss army knife of byte manipulation
          • Pip3line is a raw bytes manipulation utility, able to apply well known and less well known transformations from anywhere to anywhere (almost).
        • dnstwist
          • Domain name permutation engine for detecting typo squatting, phishing and corporate espionage
      • DNS
        • DNSChef
          • DNSChef is a highly configurable DNS proxy for Penetration Testers and Malware Analysts. A DNS proxy (aka "Fake DNS") is a tool used for application network traffic analysis among other uses. For example, a DNS proxy can be used to fake requests for "badguy.com" to point to a local machine for termination or interception instead of a real host somewhere on the Internet.
        • Passive DNS
          • A tool to collect DNS records passively to aid Incident handling, Network Security Monitoring (NSM) and general digital forensics. * PassiveDNS sniffs traffic from an interface or reads a pcap-file and outputs the DNS-server answers to a log file. PassiveDNS can cache/aggregate duplicate DNS answers in-memory, limiting the amount of data in the logfile without losing the essense in the DNS answer.
      • HTTP Traffic
        • Captipper
          • CapTipper is a python tool to analyze, explore and revive HTTP malicious traffic. CapTipper sets up a web server that acts exactly as the server in the PCAP file, and contains internal tools, with a powerful interactive console, for analysis and inspection of the hosts, objects and conversations found.
      • PCAPs/Packet Capture
        • CapLoader
          • CapLoader is a Windows tool designed to handle large amounts of captured network traffic. CapLoader performs indexing of PCAP/PcapNG files and visualizes their contents as a list of TCP and UDP flows. Users can select the flows of interest and quickly filter out those packets from the loaded PCAP files. Sending the selected flows/packets to a packet analyzer tool like Wireshark or NetworkMiner is then just a mouse click away.
        • Netdude
          • The Network Dump data Displayer and Editor is a framework for inspection, analysis and manipulation of tcpdump trace files. It addresses the need for a toolset that allows easy inspection, modification, and creation of pcap/tcpdump trace files. Netdude builds on any popular UNIX-like OS, such as Linux, the BSDs, or OSX.
        • Stenographer
          • Stenographer is a full-packet-capture utility for buffering packets to disk for intrusion detection and incident response purposes. It provides a high-performance implementation of NIC-to-disk packet writing, handles deleting those files as disk fills up, and provides methods for reading back specific sets of packets quickly and easily.
        • PCAPDB
          • PcapDB is a distributed, search-optimized open source packet capture system. It was designed to replace expensive, commercial appliances with off-the-shelf hardware and a free, easy to manage software system. Captured packets are reorganized during capture by flow (an indefinite length sequence of packets with the same src/dst ips/ports and transport proto), indexed by flow, and searched (again) by flow. The indexes for the captured packets are relatively tiny (typically less than 1% the size of the captured data).
        • Network Miner
          • NetworkMiner is a Network Forensic Analysis Tool (NFAT) for Windows (but also works in Linux / Mac OS X / FreeBSD). NetworkMiner can be used as a passive network sniffer/packet capturing tool in order to detect operating systems, sessions, hostnames, open ports etc. without putting any traffic on the network. NetworkMiner can also parse PCAP files for off-line analysis and to regenerate/reassemble transmitted files and certificates from PCAP files.
        • SilLK
          • Silk
            • The SiLK analysis suite is a collection of command-line tools for processing SiLK Flow records created by the SiLK packing system. These tools read binary files containing SiLK Flow records and partition, sort, and count these records. The most important analysis tool is rwfilter, an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options. The tools are intended to be combined in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of the tools.
          • Administering/Installing SiLK
          • SiLK Tool Tips
          • SiLK Reference Guide
          • SiLK Toolsuite Quick Reference Guide
          • flowbat
            • Awesome flow tool, SiLK backend
      • ShellCode Analysis
        • Shellcode Analysis Pipeline
          • I recently required an automated way of analyzing shellcode and verifying if it is detected by Libemu, Snort, Suricata, Bro, etc. Shellcode had to come from public sources like Shell-Storm, Exploit-DB and Metasploit. I needed an automated way of sourcing shellcode from these projects and pass it on to the analysis engines in a pipeline-like mechanism. This posts documents the method I used to complete this task and the overall progress of the project.
  • OSQuery
  • Linux
  • macOS/OS X
  • Windows

Detection Engineering

Threat Hunting

Threat Analysis

  • Tools
    • Danger-Zone
      • Correlate data between domains, IPs and email addresses, present it as a graph and store everything into Elasticsearch and JSON files.

Data Storage & Analysis Stacks