Open Source Intelligence

Table of Contents

Sort

  • Midasearch.org http://computercrimeinfo.com/cleaningid.html http://toddington.com/resources/ www.osintinsight.com/shared.php?user=Mediaquest&folderid=0\

  • OSINT - onstrat

  • Hunting Pastebin with PasteHunter

  • OSRFramework

    • OSRFramework is a GNU AGPLv3+ set of libraries developed by i3visio to perform Open Source Intelligence tasks. They include references to a bunch of different applications related to username checking, DNS lookups, information leaks research, deep web search, regular expressions extraction and many others. At the same time, by means of ad-hoc Maltego transforms, OSRFramework provides a way of making these queries graphically as well as several interfaces to interact with like OSRFConsole or a Web interface.
  • Add list of Sources:

  • UCC - Uniform Commercial Code; DOC - Current Industrial Patents; DMV - Vehicle Ownership applications; Patents - Patent DBs; Operating Licenses/Permits; Trade Journals;

  • gasmask

    • All in one Information gathering tool - OSINT

End Sort


General


OSINT Tools/Resources

  • Tools
    • blacksheepwall
      • blacksheepwall is a hostname reconnaissance tool
    • Creepy.py
      • Description: Creepy is a geolocation OSINT tool. Gathers geolocation related information from online sources, and allows for presentation on map, search filtering based on exact location and/or date, export in csv format or kml for further analysis in Google Maps.
    • Maltego
      • Description: What you use to tie everything together.
    • OpenRefine
      • Description: OpenRefine is a power tool that allows you to load data, understand it, clean it up, reconcile it to master database, and augment it with data coming from Freebase or other web sources. All with the comfort and privacy of your own computer.
    • Oryon C Portable
      • Oryon C Portable is a web browser designed to assist researchers in conducting Open Source Intelligence investigations. Oryon comes with dozens of pre-installed tools and a select set of links cataloged by category – including those that can be found in the OI Shared Resources.
    • OSINT Mantra
    • Recon-ng
      • Description: Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly.
    • TouchGraph SEO Browser
      • Use this free Java application to explore the connections between related websites.

Company/People Searching

  • data.com
  • LittleSis
    • LittleSis is a free database of who-knows-who at the heights of business and government.
  • Jigsaw
    • Jigsaw is a prospecting tool used by sales professionals, marketers and recruiters to get fresh and accurate sales leads and business contact information.
  • Spokeo
    • Spokeo is a people search engine that organizes white pages listings, public records and social network information into simple profiles to help you safely find and learn about people.\
  • Hoovers
    • Search over 85 million companies within 900 industry segments; Hoover's Reports Easy-to-read reports on key competitors, financials, and executives
  • Market Visual
    • Search Professionals by Name, Company or Title
  • Glass Door
    • Search jobs then look inside. Company salaries, reviews, interview questions, and more all posted anonymously by employees and job seekers.
  • 192
    • Find people, businesses and places in the UK with 192.com. Directory enquiries, a people finder, business listings and detailed maps with aerial photos.
  • corporationwiki
  • orbis
    • Company information across the globe

CVS/Git/Similar Focused

  • repo-supervisor
  • GitPrey
    • GitPrey is a tool for searching sensitive information or data according to company name or key word something.The design mind is from searching sensitive data leakling in Github:
  • git-all-secrets
    • A tool to capture all the git secrets by leveraging multiple open source git searching tools
  • github-firehose
  • Gitem
    • Gitem is a tool for performing Github organizational reconnaissance.
  • Truffle Hog
    • Searches through git repositories for high entropy strings, digging deep into commit history and branches. This is effective at finding secrets accidentally committed that contain high entropy.
  • dvcs-ripper
    • Rip web accessible (distributed) version control systems: SVN, GIT, Mercurial/hg, bzr, ... It can rip repositories even when directory browsing is turned off.
  • Truffle Hog
    • Searches through git repositories for high entropy strings, digging deep into commit history
  • DVCS-Pillage
    • Pillage web accessible GIT, HG and BZR repositories. I thought it would be useful to automate some other techniques I found to extract code, configs and other information from a git,hg, and bzr repo's identified in a web root that was not 100% cloneable. Each script extracts as much knowledge about the repo as possible through predictable file names and known object hashes, etc.
  • gitdigger
    • gitDigger: Creating realworld wordlists from github hosted data.
  • gitrob
    • Gitrob is a command line tool which can help organizations and security professionals find sensitive information lingering in publicly available files on GitHub. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information. Looking for sensitive information in GitHub repositories is not a new thing, it has been known for a while that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.
  • reposcanner
    • Python script to scan Git repos for interesting strings

DNS Stuff
  • dauntless
    • Tools for analysing the forward DNS data set published at https://scans.io/study/sonar.fdns_v2
  • dnstwist
    • Domain name permutation engine for detecting typo squatting, phishing and corporate espionage
  • typofinder
    • Typofinder for domain typo discovery

Email Gathering/Reconnaissance

  • Articles/Writeups
  • Tools
    • SimplyEmail
      • What is the simple email recon tool? This tool was based off the work of theHarvester and kind of a port of the functionality. This was just an expansion of what was used to build theHarvester and will incorporate his work but allow users to easily build Modules for the Framework. Which I felt was desperately needed after building my first module for theHarvester.
    • Email Reconnaissance and Phishing Template Generation Made Simple
    • theHarvester
      • theHarvester is a tool for gathering e-mail accounts, subdomain names, virtual hosts, open ports/ banners, and employee names from different public sources (search engines, pgp key servers).
    • discover.sh
      • For use with Kali Linux. Custom bash scripts used to automate various pentesting tasks.
    • Cr3dOv3r
      • Cr3dOv3r simply you give it an email then it does two simple jobs (but useful): Search for public leaks for the email and if it any, it returns with all available details about the leak (Using hacked-emails site API). Now you give it this email's old or leaked password then it checks this credentials against 16 websites (ex: facebook, twitter, google...) then it tells you if login successful in any website!

Fancy Search Engines

  • Entity Cube
    • EntityCube is a research prototype for exploring object-level search technologies, which automatically summarizes the Web for entities (such as people, locations and organizations) with a modest web presence.
  • Silobreaker
    • Enterprise Semantic Search Engine, allows virtualisation of data, analytics and exploration of key data.
  • iSeek
    • Another handy search engine that break results down into easy to manage categories.
  • Carrot2
    • Carrot2 organizes your search results into topics. With an instant overview of what's available, you will quickly find what you're looking for.
  • Sqoop
    • OSINT search engine of public documents(handy)
  • GlobalFileSearch * An FTP Search Engine that may come in handy.
  • NAPALM FTP Indexer

General Meta Data

  • Just-Metadata
    • Just-Metadata is a tool that can be used to gather intelligence information passively about a large number of IP addresses, and attempt to extrapolate relationships that might not otherwise be seen. Just-Metadata has "gather" modules which are used to gather metadata about IPs loaded into the framework across multiple resources on the internet. Just-Metadata also has "analysis" modules. These are used to analyze the data loaded Just-Metadata and perform various operations that can identify potential relationships between the loaded systems.
  • MetaGooFil
    • Description: Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target company. The tool will perform a search in Google to identify and download the documents to local disk and then will extract the metadata with different libraries like Hachoir, PdfMiner and others. With the results it will generate a report with usernames, software versions and servers or machine names that will help Penetration testers in the information gathering phase.
  • Metashield Analyzer
    • Description: Metadata documents can help a malicious user to obtain information that is beyond our control in an enterprise environment. Metashield Analyzer is an online service that allows easily check if your office documents contain metadata.
  • PowerMeta
    • PowerMeta searches for publicly available files hosted on various websites for a particular domain by using specially crafted Google, and Bing searches. It then allows for the download of those files from the target domain. After retrieving the files, the metadata associated with them can be analyzed by PowerMeta. Some interesting things commonly found in metadata are usernames, domains, software titles, and computer names.

General Data Scrapers

  • XRAY
    • XRay is a tool for recon, mapping and OSINT gathering from public networks.
  • NameCheck
    • Search usernames across multiple services/domain registries
  • [TheHarvester](From: https://code.google.com/p/theharvester/)
    • Description: The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database. This tool is intended to help Penetration testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization.
  • OSINT OPSEC Tool
    • Description: The OSINT OPSEC Tool monitors multiple 21st Century OSINT sources real-time for keywords, then analyses the results, generates alerts, and maps trends of the data, finding all sorts of info people probably don't want others to see...
  • Pattern
    • Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google,; Twitter, Wikipedia), web crawler, HTML DOM parser; Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet; Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron); Network Analysis: graph centrality and visualization.

Google Hacking

  • Google Hacking for Penetration Testers
  • ExpoitDB archive of the google hacking database
  • Google Hacking Database
    • We call them 'googledorks': Inept or foolish people as revealed by Google. Whatever you call these fools, you've found the center of the Google Hacking Universe!
  • Google Hacking - Search Diggity tool
    • SearchDiggity 3.1 is the primary attack tool of the Google Hacking Diggity Project. It is Bishop Fox’s MS Windows GUI application that serves as a front-end to the most recent versions of our Diggity tools: GoogleDiggity, BingDiggity, Bing LinkFromDomainDiggity, CodeSearchDiggity, DLPDiggity, FlashDiggity, MalwareDiggity, PortScanDiggity, SHODANDiggity, BingBinaryMalwareSearch, and NotInMyBackYard Diggity.
  • GoogD0rker
    • GoogD0rker is a tool for firing off google dorks against a target domain, it is purely for OSINT against a specific target domain. Designed for OSX originally however googD0rker txt now works on all nix platforms.

Network Information Search Engines

  • Whoisology
    • Whoisology is a domain name ownership archive with literally billions of searchable and cross referenced domain name whois records.

Site Specific
  • AWS
    • AWSBucketDump
      • AWSBucketDump is a tool to quickly enumerate AWS S3 buckets to look for loot. It's similar to a subdomain bruteforcer but is made specifically for S3 buckets and also has some extra features that allow you to grep for delicious files as well as download interesting files if you're not afraid to quickly fill up your hard drive.
  • Facebook
    • pymk-inspector
      • The pymk-inspector is a tool built by Gizmodo's Special Projects Desk that we used for our investigation into Facebook's people you may know (pymk) algorithm.
  • Github
  • LinkedIn
    • InSpy
      • A LinkedIn enumeration tool
    • linkedin
      • Linkedin Scraper using Selenium Web Driver, Firefox 45, Ubuntu and Scrapy
    • LinkedInt: A LinkedIn scraper for reconnaissance during adversary simulation
    • LinkedIn Gatherer
    • socilab
      • This site allows users to visualize and analyze their LinkedIn network using methods derived from social-scientific research. Full sample output is shown here. The site is free and open-source. Have fun!
    • Linkedin_profiles
      • This script uses selenium to scrape linkedin employee details from a specified company. If the script isn't working, you can always browse to the desired company's employee page and paste in the link on line 69 like this: "employees_page = url"
    • The Secrets of LinkedIn
      • Grabbing usernames/connections(link analysis)
    • The Endorser
      • An OSINT tool that allows you to draw out relationships between people on LinkedIn via endorsements/skills.
  • Tinder
  • Twitter
    • OneMillionTweetMap
      • This page maps the last geolocalized tweets delivered by the twitter stream API. ... YES - IN REAL-TIME - and we keep "only" the last one million tweets.
    • tweets_analyzer
      • Tweets metadata scraper & activity analyzer
    • Tweet Archivist
    • tweets_analyzer
      • Tweets metadata scraper & activity analyzer
    • Tinfoleak
      • tinfoleak is a simple Python script that allow to obtain: basic information about a Twitter user (name, picture, location, followers, etc.); devices and operating systems used by the Twitter user; applications and social networks used by the Twitter user; place and geolocation coordinates to generate a tracking map of locations visited; show user tweets in Google Earth!; download all pics from a Twitter user; hashtags used by the Twitter user and when are used (date and time); user mentions by the the Twitter user and when are occurred (date and time); topics used by the Twitter user

Social Media Search/Enumeration

  • CheckUsernames
    • Check the use of your brand or username on 160 Social Networks
  • NameCHK
    • Check to see if your desired username or vanity url is still available at dozens of popular Social Networking and Social Bookmarking websites.
  • Scythe
    • The ability to test a range of email addresses across a range of sites (e.g. social media, blogging platforms, etc...) to find where those targets have active accounts. This can be useful in a social engineering test where you have email accounts for a company and want to list where these users have used their work email for 3rd party web based services.
  • Social Mention
    • Social Mention is a social media search engine that searches user-generated content such as blogs, comments, bookmarks, events, news, videos, and more
  • Whos Talkin
    • social media search tool that allows users to search for conversations surrounding the topics that they care about most.