Clear Web OSINT Suite

Source: https://github.com/N1ckw1ck/Clear_OSINT

A set of Python command-line tools for open-source intelligence gathering on the clearnet. Phone numbers, domains, IP addresses, and image metadata — each tool runs standalone and outputs a structured terminal report with an optional JSON save.

No accounts required to get started. Some tools support optional API keys for enriched data, but all produce useful output without them.

Tools

Phone OSINT
c_phone_osint.py
Number validation, carrier detection, spam likelihood, geocoding, and optional clearnet mention scan across Google and DuckDuckGo.
Domain OSINT
c_url_osint.py
DNS, WHOIS, SSL, HTTP headers, technology fingerprint, risk score, sensitive path probe, WordPress scan, and optional page crawl.
IP OSINT
c_ip_osint.py
Geolocation, ASN and BGP routing, passive DNS, reverse IP domain discovery, Tor exit check, host classification, and optional port probe.
EXIF Tool
exif_tool.py
Read, display, strip, and write EXIF metadata on images. GPS coordinates, timestamps, device info — with backup and batch support.

Quick Start

  1. Clone the repository or copy the desired .py files into a folder.
  2. Create and activate a Python virtual environment inside that folder.
  3. Install dependencies from requirements.txt.
  4. Run the tool.
bash
pip install -r requirements.txt
python c_phone_osint.py
python c_url_osint.py
python c_ip_osint.py
python exif_tool.py
⚠ Disclaimer These tools are for authorized and ethical use only. You are responsible for compliance with all applicable laws and terms of service. The author assumes no liability for misuse.

Overview

The Clear Web OSINT Suite is a collection of standalone Python tools for clearnet intelligence gathering. Each script is self-contained — download only what you need. All use free or freemium data sources, with optional paid API key support for enriched output.

README: https://github.com/N1ckw1ck/Clear_OSINT/blob/main/README.md

Prerequisites

  • Python 3.11+
  • pip and a virtual environment (recommended)
  • Internet access — all tools make outbound requests to public APIs and data sources

Install all dependencies

bash
pip install -r requirements.txt
requirements.txt
# All / most tools
requests
phonenumbers

# c_phone_osint.py
serpapi

# c_url_osint.py
beautifulsoup4
dnspython
python-whois

# exif_tool.py
Pillow
piexif
Note If you are only using one tool, install only its dependencies rather than the full requirements.txt. Each tool's install section lists exactly what it needs.

Phone OSINT

Phone number intelligence tool. Validates and parses numbers using the phonenumbers library, checks spam likelihood, geocodes the carrier region to approximate coordinates, and optionally scans the clearnet for mentions of the number in multiple formats.

Setup & API Keys

The tool works without any API keys. Two optional keys will enrich the report with additional data:

c_phone_osint.py
# IPQualityScore — fraud score, DNC status, carrier enrichment
# Sign up free at ipqualityscore.com
IPQS_API_KEY: str = ''

# SerpAPI — clearnet mention scan via Google + DuckDuckGo
# Sign up free at serpapi.com (100 searches/month)
SERPAPI_KEY: str = ''

Install

bash
pip install requests phonenumbers serpapi

Usage

bash
python c_phone_osint.py
python c_phone_osint.py --save   # auto-save report to JSON

Usage flow

  1. Select US/Canada or International, then enter the number in prompted parts.
  2. Main report generates immediately — validation, carrier, region, formats, spam check, geocoordinates, and IPQS enrichment if a key is set.
  3. Prompted to run a clearnet mention scan (requires SerpAPI key). Default is no.
  4. If the scan returns more than 10 results, prompted to save all to JSON.

How It Works

  • phonenumbers library parses validity, country, region, line type, carrier, and timezone.
  • SkipCalls (no key) checks spam likelihood and report count.
  • Nominatim / OpenStreetMap (no key) geocodes the carrier region to approximate coordinates — city or state level, not exact.
  • IPQualityScore (optional key) adds fraud score, DNC registry status, prepaid flag, recent abuse flag, VOIP flag, MCC/MNC, and CNAM lookup.
  • SerpAPI (optional key) searches Google and DuckDuckGo for the number in E.164, national, international, and digits-only formats, deduplicates results, and categorises each hit — scam report, data broker, social media, business directory, or general web.

Notes

  • Spam likelihood via SkipCalls produces false negatives, making it a signal, not a definitive verdict.
  • Carrier detection is unreliable for US numbers specifically. Paid services do better.
  • Geocoding is region/city level, not a precise location.
  • IPQS can be troublesome to sign up for. It is advised to do so without a VPN and from a standard browser. Their duplicate account detection is aggressive.
  • SerpAPI's free tier allows 100 searches per month. The mention scan consumes 2 per run (one per engine).

Domain OSINT

Comprehensive domain intelligence tool. Resolves DNS records, fetches WHOIS data, parses the SSL certificate, analyses HTTP headers and security posture, fingerprints web technologies, probes sensitive paths, optionally scans for WordPress vulnerabilities, checks email security records, and crawls the site for emails and external domain references.

Install

bash
pip install requests beautifulsoup4 dnspython python-whois

Usage

bash
python c_url_osint.py
python c_url_osint.py --save   # auto-save report to JSON

Usage flow

  1. Enter the target domain (with or without https://) and a max page crawl limit (1–50, default 20).
  2. DNS, WHOIS, SSL, HTTP headers, tech fingerprint, CSP quality, HTTP method enumeration, email security (SPF / DMARC / DKIM), and sensitive path probe run automatically.
  3. Prompted to run an optional URL threat scan (scrapes IPQS public page).
  4. Prompted to run an optional port scan against the resolved IP — ensure you are authorized.
  5. Prompted to run an optional page crawl up to your specified limit.
  6. Prompted to save the full report to JSON.

How It Works

  • Resolves A, PTR, MX, NS, and TXT records via dnspython.
  • Fetches WHOIS registration data including registrar, creation and expiry dates, and nameservers.
  • Retrieves and parses the SSL/TLS certificate — issuer, expiry, days remaining, Subject Alternative Names.
  • Fetches HTTP response headers and checks for the presence or absence of seven key security headers with vulnerability context.
  • Probes a list of sensitive paths.git/HEAD, .env, phpinfo.php, WordPress endpoints, and more — and reports status codes and content hints.
  • Evaluates CSP header quality — flags unsafe-inline, unsafe-eval, wildcard sources, and missing directives.
  • Fingerprints web technologies from headers and HTML — server, CMS, JS framework, CDN, and hosting provider.
  • If WordPress is detected, extracts version from up to four sources and maps it against known CVEs.
  • Checks SPF, DMARC, and DKIM selectors via DNS TXT lookups with policy grading.
  • Calculates a passive risk score (0–100) from already-collected data — no extra requests.
  • Optionally probes 14 ports via direct TCP connect with banner grabbing on text-protocol ports.
  • Optionally crawls the domain with a 1-second delay between pages, collecting emails, internal links, and external domain references.

Notes

  • The crawl uses a 1-second delay between requests out of politeness. On large sites, the page cap will be reached — raise it carefully.
  • Technology fingerprinting is based on headers and static HTML. Client-side-rendered sites may not reveal their full stack this way.
  • The URL threat scan scrapes a public page rather than using an API. It may break if the page structure changes.
  • WHOIS data availability varies by TLD. Privacy-protected domains return minimal registrant information.
  • The risk score is a passive indicator derived from your scan data, not a dedicated threat intelligence feed.
  • As of April 2026, the tool can scrape sites behind Cloudflare and similar CDNs for email and link extraction.

IP OSINT

IP address intelligence tool. Covers geolocation, ASN and BGP routing data, passive DNS, DNS chain resolution, hosted domain discovery via reverse IP lookup and TLS certificate SANs, Tor exit node check, host classification, and optional ICMP ping and port scan.

Setup & API Keys

The tool works without any API keys. Two optional keys unlock additional data:

c_ip_osint.py
# AbuseIPDB — abuse confidence score, report count, usage type
# Free tier available at abuseipdb.com
ABUSEIPDB_API_KEY: str = ''

# Shodan — open ports, service banners, hostnames, OS, tags
# Requires paid membership for /shodan/host/{ip} endpoint
SHODAN_API_KEY: str = ''

Install

bash
pip install requests

Usage

bash
python c_ip_osint.py
python c_ip_osint.py --save   # auto-save report to JSON

Usage flow

  1. Enter a target IPv4 or IPv6 address.
  2. Private or loopback IPs trigger a local interface scan and exit.
  3. For public IPs, geolocation, ASN, routing, passive DNS, DNS chain, domain discovery, Tor check, and host classification run automatically. AbuseIPDB and Shodan enrichment run if keys are set.
  4. Prompted to run an optional ICMP ping (10 packets).
  5. Prompted to run an optional common port probe (24 ports, direct TCP connect).
  6. Prompted to save the full report to JSON.

Test target

The Nmap project maintains a publicly scannable host at 45.33.32.156 (scanme.nmap.org) you can use to test the tool.

How It Works

  • Classifies the IP (public/private/loopback, IPv4/IPv6) and performs a reverse DNS lookup.
  • Fetches geolocation, timezone, and ASN data from ipapi.co — 1000 requests/day free, no key required.
  • Queries RIPEstat for BGP prefix, announcement status, RIR allocation, ASN holder, abuse contacts, and routing peers.
  • Fetches passive DNS records from RIPEstat showing domain names historically resolved to the target IP.
  • Resolves a recursive DNS chain from the reverse DNS hostname.
  • Performs reverse IP domain discovery by querying HackerTarget, extracting TLS certificate SANs, cross-referencing Shodan and passive DNS, then HTTP-probing each discovered domain with a Host header to confirm live services.
  • Checks the IP against the Tor Project's bulk exit node list.
  • Classifies the host as datacenter/cloud, residential ISP, or mobile carrier based on ASN name, org string, and BGP prefix size.
  • Optionally sends ICMP ping via the system ping binary and parses RTT and packet loss statistics.
  • Optionally probes 24 common ports concurrently via direct TCP connect, with banner grabbing on text-protocol ports.

Notes

  • HackerTarget reverse IP lookup has a limit of approximately 100 requests per day per IP. The geocoding via ipapi.co allows 1000 per day.
  • Geolocation precision varies significantly by IP and provider. ISP-owned blocks may only resolve to country or region level.
  • Passive DNS coverage depends on what RIPEstat has observed. Many IPs return no records.
  • The port probe uses short TCP connect timeouts. Firewalls that drop rather than reject will cause ports to time out silently — false negatives are possible.
  • Shodan's free API tier does not include access to the /shodan/host/{ip} endpoint. A paid membership is required. The tool works fine without it.
  • An AbuseIPDB score of 0 does not guarantee an IP is clean. A score of 100 does not confirm it is malicious.

EXIF Tool

Reads, displays, strips, and writes EXIF metadata from image files. Supports JPEG, TIFF, PNG and most Pillow-readable formats. GPS coordinates are decoded to decimal degrees with a Google Maps link. Operates interactively or on paths passed as arguments.

Install

bash
pip install Pillow piexif

Usage

bash
python exif_tool.py                         # interactive mode
python exif_tool.py photo.jpg               # single file
python exif_tool.py photo1.jpg photo2.jpg   # multiple files

Usage flow (per file)

  1. All EXIF fields are printed — tag name alongside decoded value.
  2. Prompted to strip EXIF. y strips with a .backup copy. b strips without backup. Enter skips.
  3. Prompted to add or edit EXIF fields. Choose Merge (keeps existing, overwrites selected fields) or Wipe (clears all fields first then writes).
  4. Select which fields to set from a numbered menu, enter values, then d to apply.

Writable fields

FieldFormat
DateTimeYYYY-MM-DD HH:MM:SS or YYYY:MM:DD HH:MM:SS
ImageDescriptionFree text
ArtistFree text
CopyrightFree text
MakeDevice manufacturer
ModelDevice model
SoftwareFree text
UserCommentFree text (stored as ASCII)
GPSlat,lon — e.g. 51.5074,-0.1278

How It Works

  • Opens images with Pillow and reads raw EXIF via _getexif(). Tag IDs are resolved using PIL.ExifTags.TAGS and GPSTAGS.
  • GPS IFD entries are decoded from rational tuples to decimal degrees. A Google Maps link is constructed from the coordinates.
  • Stripping works by writing an empty piexif block, or — for formats piexif doesn't support — by copying pixel data into a new image with no EXIF.
  • Writing uses piexif to load the existing EXIF dict (or start fresh on wipe), apply updates per field, and save back to the original path.
  • A .backup copy is made before any destructive operation when requested.

Setup & Dependencies

Virtual Environment

Running in a virtual environment keeps dependencies isolated and avoids conflicts with system Python packages. Recommended for all tools.

bash
# Create a venv inside your project folder
python3 -m venv .venv

# Activate (macOS / Linux)
source .venv/bin/activate

# Activate (Windows)
.venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Deactivate when done
deactivate

API Keys

All tools run without API keys. The following optional keys extend what certain tools can return. All linked services offer a free tier.

KeyToolWhat it addsFree tier
IPQSPhone OSINTFraud score, DNC status, CNAM, VOIP/prepaid flagsYes — ipqualityscore.com
SerpAPIPhone OSINTClearnet mention scan via Google + DuckDuckGoYes — serpapi.com
AbuseIPDBIP OSINTAbuse confidence score, report count, usage typeYes — abuseipdb.com
ShodanIP OSINTOpen ports, service banners, hostnames, OS, tagsNo — paid plan required for host lookup

Keys are pasted directly into the relevant script at the top of the file where indicated. They are never sent anywhere other than the respective API endpoint.

IPQS Account Creation IPQS uses aggressive duplicate account detection. Create an account without a VPN active and from a standard browser (not Tor, not a public network). If you already have an account and are having trouble accessing the API, this is likely why.

Rate Limits

The tools rely on several free public APIs and data sources. Keep the following limits in mind when running scans repeatedly.

SourceUsed byLimit
ipapi.coIP OSINT1,000 requests / day (no key)
HackerTargetIP OSINT~100 requests / day per IP
NominatimPhone OSINT1 request / second — do not batch
RIPEstatIP OSINTReasonable use, no hard limit
SerpAPIPhone OSINT100 searches / month (free tier)
AbuseIPDBIP OSINT1,000 checks / day (free tier)
SkipCallsPhone OSINTReasonable use, no published limit

Disclaimer & Use

  • These tools are for authorized and ethical use only. You are responsible for ensuring your use complies with all applicable laws, regulations, and terms of service.
  • Port scanning an IP you do not own or have explicit permission to scan is illegal in many jurisdictions. The IP OSINT tool prompts before port scanning, make sure you are authorized.
  • Crawling or scraping websites may violate their terms of service. The domain tool prompts before crawling.
  • Requests made by these tools are potentially traceable back to your IP. If you require anonymity for your research, route through a VPN before running any tool.
  • The author does not assume liability for any misuse of these tools.