Open Source Intelligence
Table of Contents
- Presentations & Talks
- DNS Stuff/related
- Email Gathering
- Fancy Search Engines
- Search Engine Dorks
- Site Specific Tools
- Social Media Search/Enumeration
- Company/People Searching
- Reference Sites
- Add list of Sources:
- UCC - Uniform Commercial Code;
- DOC - Current Industrial Patents;
- DMV - Vehicle Ownership applications;
- Patents - Patent DBs;
- Operating Licenses/Permits;
- Trade Journals;
- SWOT - Strengths, Weaknesses, Opportunities, Threats
- Hunting Pastebin with PasteHunter
- Open Source Intelligence Gathering 101 - appseco.com
- Open Source Intelligence Gathering 201 - appseco.com
- Open Source Intelligence Gathering: Techniques, Automation, and Visualization - Christopher Maddalena
- The OSINT Connection: Intelligence In Executive Protection - protectioncircle.com
- See what are the popular related topics people are searching for. This will help widen your search scope.
- Google Alerts are email updates of the latest relevant Google results (web, news, etc.) based on your queries.
- PasteLert is a simple system to search pastebin.com and set up alerts (like google alerts) for pastebin.com entries. This means you will automatically recieve email whenever your term(s) is/are found in new pastebin entries!
- Google Trends
OSINT Based News
- Just Security is an online forum for the rigorous analysis of U.S. national security law and policy. We aim to promote principled and pragmatic solutions to national security problems that decision-makers face. Our Board of Editors includes individuals with significant government experience, civil society attorneys, academics, and other leading voices. Just Security is based at the Center for Human Rights and Global Justice at New York University School of Law.
- By and for citizen investigative journalists
- NightWatch is an executive commentary and analysis of events that pose or advance threats to US national security interests. It is deliberately edgy in the interest of clarity and brevity. As a product for executives, the distribution and all feedback comments are anonymous.
- RSOE EDIS - Emergency and Disaster Information Service
- OSINT Framework
- OSINT Resources - greynetwork2
- Intel Techniques - Links
- toddington - resources
- onstrat - osint
Open Source Intelligence (OSINT) Tools & Resources - osint.link
- Seems pretty good.
- IntelTechniques OSINT Flowcharts
- Fantastic OSINT and where to find it - blindseeker/malware focused
- Some blog posts describing/bringing you up to speed on OSINT by krypt3ia
- Glass Reflections in Pictures + OSINT = More Accurate Location
- Exploring the Github Firehose
- OSINT Through Sender Policy Framework (SPF) Records
- Hunting with ꓘamerka 2.0 aka FIST (Flickr, Instagram, Shodan, Twitter)
ꓘamerka 2.0 aka FIST (Flickr, Instagram, Shodan, Twitter)
- Build interactive map of cameras, printers, tweets and photos. The script creates a map of cameras, printers, tweets and photos based on your coordinates. Everything is clearly presented in form of interactive map with icons and popups.
Talks & Presentations
- Cognitive Bias and Critical Thinking in Open Source Intelligence - Defcamp 2014
- Dark Arts of OSINT Skydogcon
Developing a Open Source Threat Intelligence Program—Edward McCabe
- What if you could get out in front of common threats such as botnets, scanners and malware? Good news, you can. Learn about one geeks struggle with life on the Internet of (bad) things when it comes to being online, identifying “odd” things, and developing an Open Source Threat Intelligence Program from Open Source Tools and Public Sources.
- Corporate Espionage: Gathering Actionable Intelligence Via Covert Operations - Brent White - Defcon22
- How to Use Python to Spy on Your Friends: Web APIs, Recon ng, & OSINT
Practical OSINT - Shane MacDougall
- There’s more to life to OSINT than google scraping and social media harvesting. Learn some practical methods to automate information gathering, explore some of the most useful tools, and learn how to recognize valuable data when you see it. Not only will we explore various tools, attendees will get access to unpublished transforms they can use/modify for their own use.
- Pwning People Personally - Josh Schwartz
You're Leaking Trade Secrets - Defcon22 Michael Schrenk
- Networks don't need to be hacked for information to be compromised. This is particularly true for organizations that are trying to keep trade secrets. While we hear a lot about personal privacy, little is said in regard to organizational privacy. Organizations, in fact, leak information at a much greater rate than individuals, and usually do so with little fanfare. There are greater consequences for organizations when information is leaked because the secrets often fall into the hands of competitors. This talk uses a variety of real world examples to show how trade secrets are leaked online, and how organizational privacy is compromised by seemingly innocent use of The Internet.
- ZOMG Its OSINT Heaven Tazz Tazz
- blacksheepwall is a hostname reconnaissance tool
- Description: What you use to tie everything together.
Oryon C Portable
- Oryon C Portable is a web browser designed to assist researchers in conducting Open Source Intelligence investigations. Oryon comes with dozens of pre-installed tools and a select set of links cataloged by category – including those that can be found in the OI Shared Resources.
- OSINT Mantra
- Description: Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly.
TouchGraph SEO Browser
- Use this free Java application to explore the connections between related websites.
- Tool that automates OSINT collection. Seems to gather from a variety of sources. Perl script.
- All in one Information gathering tool - OSINT
- Correlate data between domains, ips and email addresses, present it as a graph and store everything into Elasticsearch and JSON files.
- Description: OpenRefine is a power tool that allows you to load data, understand it, clean it up, reconcile it to master database, and augment it with data coming from Freebase or other web sources. All with the comfort and privacy of your own computer.
- OSRFramework is a GNU AGPLv3+ set of libraries developed by i3visio to perform Open Source Intelligence tasks. They include references to a bunch of different applications related to username checking, DNS lookups, information leaks research, deep web search, regular expressions extraction and many others. At the same time, by means of ad-hoc Maltego transforms, OSRFramework provides a way of making these queries graphically as well as several interfaces to interact with like OSRFConsole or a Web interface.
- Description: Creepy is a geolocation OSINT tool. Gathers geolocation related information from online sources, and allows for presentation on map, search filtering based on exact location and/or date, export in csv format or kml for further analysis in Google Maps.
- Research Collection/Organization
- LittleSis is a free database of who-knows-who at the heights of business and government.
- Jigsaw is a prospecting tool used by sales professionals, marketers and recruiters to get fresh and accurate sales leads and business contact information.
- Spokeo is a people search engine that organizes white pages listings, public records and social network information into simple profiles to help you safely find and learn about people.\
- Search over 85 million companies within 900 industry segments; Hoover's Reports Easy-to-read reports on key competitors, financials, and executives
- Search Professionals by Name, Company or Title
- Search jobs then look inside. Company salaries, reviews, interview questions, and more all posted anonymously by employees and job seekers.
- Find people, businesses and places in the UK with 192.com. Directory enquiries, a people finder, business listings and detailed maps with aerial photos.
- Company information across the globe
- Country Specific Resources
- GitPrey is a tool for searching sensitive information or data according to company name or key word something.The design mind is from searching sensitive data leakling in Github:
- A tool to capture all the git secrets by leveraging multiple open source git searching tools
- A library that will connect to github and emit events from the Github Event API in near-real-time
- Exploring the Github Firehose
- Gitem is a tool for performing Github organizational reconnaissance.
- Searches through git repositories for high entropy strings, digging deep into commit history and branches. This is effective at finding secrets accidentally committed that contain high entropy.
- Rip web accessible (distributed) version control systems: SVN, GIT, Mercurial/hg, bzr, ... It can rip repositories even when directory browsing is turned off.
- Searches through git repositories for high entropy strings, digging deep into commit history
- Pillage web accessible GIT, HG and BZR repositories. I thought it would be useful to automate some other techniques I found to extract code, configs and other information from a git,hg, and bzr repo's identified in a web root that was not 100% cloneable. Each script extracts as much knowledge about the repo as possible through predictable file names and known object hashes, etc.
- gitDigger: Creating realworld wordlists from github hosted data.
- Gitrob is a command line tool which can help organizations and security professionals find sensitive information lingering in publicly available files on GitHub. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information. Looking for sensitive information in GitHub repositories is not a new thing, it has been known for a while that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.
- Python script to scan Git repos for interesting strings
- Searches full repo history for secrets and keys
- Reposcanner is a python script to search through the commit history of Git repositories looking for interesting strings such as API keys, inspired by truffleHog.
- DNS Stuff
- Waybackpack is a command-line tool that lets you download the entire Wayback Machine archive for a given URL.
domain - jhaddix
- Recon-ng and Alt-DNS are awesome. This script combines the power of these tools with the ability to run multiple domains within the same session. TLDR; I just want to do my subdomain discovery via ONE command and be done with it. Only 1 module needs an api key (/api/google_site) find instructions for that on the recon-ng wiki. Script to enumerate subdomains, leveraging recon-ng. Uses google scraping, bing scraping, baidu scraping, yahoo scraping, netcraft, and bruteforces to find subdomains. Plus resolves to IP
- checkO365 is a tool to check if a target domain is using O365
- What is the simple email recon tool? This tool was based off the work of theHarvester and kind of a port of the functionality. This was just an expansion of what was used to build theHarvester and will incorporate his work but allow users to easily build Modules for the Framework. Which I felt was desperately needed after building my first module for theHarvester.
- Email Reconnaissance and Phishing Template Generation Made Simple
- theHarvester is a tool for gathering e-mail accounts, subdomain names, virtual hosts, open ports/ banners, and employee names from different public sources (search engines, pgp key servers).
- For use with Kali Linux. Custom bash scripts used to automate various pentesting tasks.
- Cr3dOv3r simply you give it an email then it does two simple jobs (but useful): Search for public leaks for the email and if it any, it returns with all available details about the leak (Using hacked-emails site API). Now you give it this email's old or leaked password then it checks this credentials against 16 websites (ex: facebook, twitter, google...) then it tells you if login successful in any website!
- Infoga is a tool gathering email accounts informations (ip,hostname,country,...) from different public source (search engines, pgp key servers and shodan) and check if emails was leaked using haveibeenpwned.com API. Is a really simple tool, but very effective for the early stages of a penetration test or just to know the visibility of your company in the Internet.
Facial Mapping Data
- Social Mapper is a Open Source Intelligence Tool that uses facial recognition to correlate social media profiles across different sites on a large scale. It takes an automated approach to searching popular social media sites for targets names and pictures to accurately detect and group a person’s presence, outputting the results into report that a human operator can quickly review.
- Social Mapper
Fancy Search Engines
- EntityCube is a research prototype for exploring object-level search technologies, which automatically summarizes the Web for entities (such as people, locations and organizations) with a modest web presence.
- Enterprise Semantic Search Engine, allows virtualisation of data, analytics and exploration of key data.
- Another handy search engine that break results down into easy to manage categories.
- Carrot2 organizes your search results into topics. With an instant overview of what's available, you will quickly find what you're looking for.
- OSINT search engine of public documents(handy)
- An FTP Search Engine that may come in handy.
- NAPALM FTP Indexer
- Entity Cube
General Meta Data
- Just-Metadata is a tool that can be used to gather intelligence information passively about a large number of IP addresses, and attempt to extrapolate relationships that might not otherwise be seen. Just-Metadata has "gather" modules which are used to gather metadata about IPs loaded into the framework across multiple resources on the internet. Just-Metadata also has "analysis" modules. These are used to analyze the data loaded Just-Metadata and perform various operations that can identify potential relationships between the loaded systems.
- Description: Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target company. The tool will perform a search in Google to identify and download the documents to local disk and then will extract the metadata with different libraries like Hachoir, PdfMiner and others. With the results it will generate a report with usernames, software versions and servers or machine names that will help Penetration testers in the information gathering phase.
- Description: Metadata documents can help a malicious user to obtain information that is beyond our control in an enterprise environment. Metashield Analyzer is an online service that allows easily check if your office documents contain metadata.
- PowerMeta searches for publicly available files hosted on various websites for a particular domain by using specially crafted Google, and Bing searches. It then allows for the download of those files from the target domain. After retrieving the files, the metadata associated with them can be analyzed by PowerMeta. Some interesting things commonly found in metadata are usernames, domains, software titles, and computer names.
General Data Scrapers
- XRay is a tool for recon, mapping and OSINT gathering from public networks.
- Search usernames across multiple services/domain registries
- [TheHarvester](From: https://code.google.com/p/theharvester/)
- Description: The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database. This tool is intended to help Penetration testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization.
OSINT OPSEC Tool
- Description: The OSINT OPSEC Tool monitors multiple 21st Century OSINT sources real-time for keywords, then analyses the results, generates alerts, and maps trends of the data, finding all sorts of info people probably don't want others to see...
- Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google,; Twitter, Wikipedia), web crawler, HTML DOM parser; Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet; Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron); Network Analysis: graph centrality and visualization.
Search Engine Dorks
- Automatically launch google hacking queries against a target domain to find vulnerabilities and enumerate a target.
Google Hacking - Search Diggity tool
- SearchDiggity 3.1 is the primary attack tool of the Google Hacking Diggity Project. It is Bishop Fox’s MS Windows GUI application that serves as a front-end to the most recent versions of our Diggity tools: GoogleDiggity, BingDiggity, Bing LinkFromDomainDiggity, CodeSearchDiggity, DLPDiggity, FlashDiggity, MalwareDiggity, PortScanDiggity, SHODANDiggity, BingBinaryMalwareSearch, and NotInMyBackYard Diggity.
- GoogD0rker is a tool for firing off google dorks against a target domain, it is purely for OSINT against a specific target domain. Designed for OSX originally however googD0rker txt now works on all nix platforms.
Network Information Search Engines
- Whoisology is a domain name ownership archive with literally billions of searchable and cross referenced domain name whois records.
- AWSBucketDump is a tool to quickly enumerate AWS S3 buckets to look for loot. It's similar to a subdomain bruteforcer but is made specifically for S3 buckets and also has some extra features that allow you to grep for delicious files as well as download interesting files if you're not afraid to quickly fill up your hard drive.
- A LinkedIn enumeration tool
- Linkedin Scraper using Selenium Web Driver, Firefox 45, Ubuntu and Scrapy
- LinkedInt: A LinkedIn scraper for reconnaissance during adversary simulation
- LinkedIn Gatherer
- This site allows users to visualize and analyze their LinkedIn network using methods derived from social-scientific research. Full sample output is shown here. The site is free and open-source. Have fun!
- This script uses selenium to scrape linkedin employee details from a specified company. If the script isn't working, you can always browse to the desired company's employee page and paste in the link on line 69 like this: "employees_page = url"
The Secrets of LinkedIn
- Grabbing usernames/connections(link analysis)
- An OSINT tool that allows you to draw out relationships between people on LinkedIn via endorsements/skills.
- this tool assists in performing reconnaissance using the LinkedIn.com website/API. Provide a search string just as you would on the original website and let ScrapedIn do all the dirty work. Output is stored as an XLSX file, however it is intended to be used with Google Spreadsheets. After importing the XLSX into Google Spreadsheets there will be a "dataset" worksheet and a "report" worksheet.
- Gathering Usernames from Google LinkedIn Results Using Burp Suite Pro - BHIS
- A Burp Suite Extension to pull Employee Names from Google and Bing LinkedIn Search Results.
- This page maps the last geolocalized tweets delivered by the twitter stream API. ... YES - IN REAL-TIME - and we keep "only" the last one million tweets.
- Tweets metadata scraper & activity analyzer
- Tweet Archivist
- Tweets metadata scraper & activity analyzer
- tinfoleak is a simple Python script that allow to obtain: basic information about a Twitter user (name, picture, location, followers, etc.); devices and operating systems used by the Twitter user; applications and social networks used by the Twitter user; place and geolocation coordinates to generate a tracking map of locations visited; show user tweets in Google Earth!; download all pics from a Twitter user; hashtags used by the Twitter user and when are used (date and time); user mentions by the the Twitter user and when are occurred (date and time); topics used by the Twitter user
- How to Find the Twitter ID from an Email Address - booleanstrings.com
- Formerly known as Tweep, Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API. Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too. Twint also makes special queries to Twitter allowing you to also scrape a Twitter user's followers, Tweets a user has liked, and who they follow without any authentication, API, Selenium, or browser emulation.
Social Media Search/Enumeration
- Check the use of your brand or username on 160 Social Networks
- Check to see if your desired username or vanity url is still available at dozens of popular Social Networking and Social Bookmarking websites.
- The ability to test a range of email addresses across a range of sites (e.g. social media, blogging platforms, etc...) to find where those targets have active accounts. This can be useful in a social engineering test where you have email accounts for a company and want to list where these users have used their work email for 3rd party web based services.
- Social Mention is a social media search engine that searches user-generated content such as blogs, comments, bookmarks, events, news, videos, and more
- social media search tool that allows users to search for conversations surrounding the topics that they care about most.