(aka OSINT for newbies, OSINT 101 or whatever else might get you interrested)

This article is meant as an introduction to OSINT for someone who is new to it. If you have previous experience it might come off as overly simple or lacking details, but hopefully there is still one or two things here for you to pick up too. I have tried to keep this brief and list very few particular tools but rather focus on topics and concepts. If I notice the article gets well received I might update it or follow up with more tool specific information later on. Enjoy!

Table of contents

1. What is OSINT?
   • Definition and explanation of OSINT
   • The importance of OSINT in the information age
2. Types of OSINT Sources
   • Social networks
   • DNS and web information
   • Government websites
   • Academic research and publications
   • News media
   • Human intelligence and interviewing
   • Steganography
3. Tools for Gathering OSINT
   • Search engines and specialized search engines
   • Social media monitoring tools
   • Scraping tools and APIs
   • Geolocation and mapping tools
   • Data visualization tools
4. Best Practices for Conducting OSINT
   • Ethics and legal considerations
   • Verification and source evaluation
   • Protecting privacy and personal information
   • Documenting and storing OSINT findings
5. Applications of OSINT
6. Conclusion

1. What is OSINT?

OSINT (open source intelligence) is originally a military term refering to collection and analysis of publicly available source. As you see it has nothing to do with open source software as one might think if coming into this from the IT sector. The sources can among others include websites, social media, news, public records and any other information that is accessible by anyone. The goal is to gather information and put it together in a way that it can be used for the desired purpose, wether it is decision-making, business or journalism. Much like a puzzle where you connect lots of smaller pieces of information to form a new full picture that is not readily apparent from the individual pieces.

In the current age where information is shared in such an abundance it has become even more important with OSINT. It can now easier than ever be used to gather insights into a broad spectrum of topics such as public opinions market trends, individual behaviour and even to identify potential threats. When getting into OSINT it is important to have a clear idea of the ethics and any laws that might apply to your subject and also the privacy imoplications when people are involved.

2. Types of OSINT sources

Open data sources; this is likely the source you will use the most when starting out with OSINT and especially so if it is for a CTF or other reasons not pertaining to real life. The list of sources will be too long to even fit here so I will rather list some examples to give an idea.

• Socal networks
  Here we have the usual suspects, Facebook, Instagram, TikTok, Vkontakte etc etc. Make sure to research local options where your target might also have a presence. Look at the scrapers for socials later in the article.
• DNS and web information
  Every registered domain has contact information in the WHOIS information. If the domain is not registered via a registrar offering anonymisation of this data it can be a good source of bits of information to feed your process. Email addresses, if you get your hands on these it is often easy to deduce if they are used at sites by trying to sign up with them. It is also often worth it to do a search in eg google on them. Also worth looking into breach databases, not to access any accounts but to glean more information on the target. Check haveIbeenpwned.com for instance. DNSdumpster.com is also a very good source to get a good idea of how a given domain looks and what it contains. Another great resource for domain related information is https://viewdns.info/. Forums ans bulletin boards often lets users have a personalized footer in their messages where a lot of information can sometimes be found.
• Government websites
  This can easily become a little bit more tricky as different countries consider different data to be public information. Again, research your target and the local options. Taxes, addresses, income and vehicle registrations are things that can be valuable assets in this area.
• Academic reasearch and publications
  This is something that can be more or less useful depending on target and angle of the investigation. Look at sites such as arXiv.org or google scholar. Also remember to look into local sites, many universities also publish papers on their own sites.
• News media
  Same here, what is the target and what is the angle? Always worth looking to gather secondary or tertiary information that can be later be biult on further if needed.
• Human intelligence - interviewing
  This is really starting to dip into another intelligence dicipline aptly named HUMINT (human intelligence). However, to a certain point it can still be considered OSINT as in this time and age it is really possible to reach out to people in a different way now.
• Steganography - hiding data in images
  While this might not be considered OSINT by some and is not commonly used in the wild, it is something you are likely to come across if your focus is playing CTFs. There is also the related topics of audio steganography and video steganography where data is hidden in audio and video files. Portswigger has a good article about these topics here.

3. Tools for gathering OSINT

• Search engines and specialized search engines
  A first start really should be to do a google search of your target as quite often it will reveal alot of other sites to further investigate. Also try search engines such as duck duck go as it does not keep you in a bubble in the same way. For good meassure have a go with bing and ..? too. Also remember there are sometimes local search engines that should be investigated. Any links you find on a target is good to also search around for. For instance if the targets profile on site A mentions 0xbadcoffee.se/myprofile, that is likely to also be mentioned on site B and so on. Reverse image search; tineye.com, google image search and yandex image search gives much information. One way I have had much success with is doing a reverse search of avatars from social media as for instance they ofter reveal other social medias where they are being used by the user.

• Social media monitoring tools
  I will not go into any detail at all here as there are just too many of these tools and I personally dont use any of them. If you are working with social media you will likely have a good idea allready. For longer term monitoring of various pages for information gathering you will have to research to find a tool that fits your needs and abilities.

• Scraping tools and APIs
  Here there are also too many tools to mention that can scrape social media with various success. Depending on the type of research you are doing you may want to look into different tools for the job. Some tools also have build in capabilities for social media collection. I still want to make specific mention the tool OSINTgram for instagram, https://tweetbeaver.com/getfriends.php and nitter.net for twitter on this topic.

• Geolocation and mapping tools
  Scrape images for EXIF data with any tool of preference, commonly mobile phones will geo tag images with the GPS coordinates when they are taken. Geolocation in itself is a rather large field and there are many techniques such as triangulation of known objects in images to determining time and date based on where shadows fall. These tasks can be done manually or more or less tool aided. This all makes it a too wide topic for this article to dive much further into. Plotting data on a map is also a wide topic that I will for now only mention, and it can be done by custom maps in Google maps or by using specialized tools. Research further to find the way you prefer to do this.

• Data visualization tools
  There are many tools to use for visualisation and most of the time it is down to personal preference, and whether or not you are doing the investigation alone or in a team that determines which tool to use. Some tools are, Maltego-ce, hmm what else? One tool that is worth checking out is https://osintcombine.tools/ where you can load multiple CSV files with up to four columns to visualize your information.

4. Best Practices for Conducting OSINT

• Ethics and legal considerations
  When collecting and analyzing information in your OSINT work it is important to be attentive to the ethical hazards. Ethics is this case is in the end up to you but just be aware of the potential integrity impacts on individuals. When is comes to the legal aspect this is something you will have to research in you geographic area as in some areas the mere act of adding PII (personally identifiable information) into databases require permission and is otherwise illegal.

• Verification and source evaluation
  Remember to value information based on how verifiable it is. DNS information is usually correct (not talking about owner information etc here but the data served) while someones post on reddit or facebook will need to be verified through other sources to be fully trustworthy.

• Protecting privacy and personal information
  While taking ethical and legal aspects into consideration also make sure that you handle the data carefully. This is very important as your dataset can contain possibly sensitive or incriminating information just as much as the analyzed final data which is perhaps more obvious. There are many ways for keeping your data secure but that is outside of the scope of this article.

• Documenting and storing OSINT findings
  This is again much down to personal preference or need based on teamwork in the same way as visualization. After analysis it can also be prudent to delete the raw material if it is not needed anymore.

5. Applications of OSINT

I will just leave this as a bullet list as there is likely little further explanation needed. These are just some examples and in no way an exhaustive list.

• Law enforcement and intelligence gathering
• crowdsourced searches for missing people or abducted children (bleeds into the above bullet but is performed by volunteers)
• Business intelligence and market research
• Investigative journalism and fact-checking
• Online reputation management and digital forensics
• investigation of disinformation campaigns
• discovery of troll farms and astroturfing
• capture the flag challenges and other entertainment

6. Conclusion

As you can see OSINT is a quite large topic that spans many disciplines and also quite easily gets you rabbit holed in several areas. Depending on your reason for getting into OSINT this can be a good or a bad thing.

For now this will have to do, if you enjoyed this article or leaned something new perhaps consider bying me a bad coffee to keep me going. Or if you are in a shopping mood have a look at the referral page for items that support the site via affiliate links at no additional cost to you!