Clear Web OSINT Suite
Source: https://github.com/N1ckw1ck/Clear_OSINT
A set of Python command-line tools for open-source intelligence gathering on the clearnet. Phone numbers, domains, IP addresses, and image metadata — each tool runs standalone and outputs a structured terminal report with an optional JSON save.
No accounts required to get started. Some tools support optional API keys for enriched data, but all produce useful output without them.
Tools
Quick Start
- Clone the repository or copy the desired
.pyfiles into a folder. - Create and activate a Python virtual environment inside that folder.
- Install dependencies from
requirements.txt. - Run the tool.
pip install -r requirements.txt python c_phone_osint.py python c_url_osint.py python c_ip_osint.py python exif_tool.py
Overview
The Clear Web OSINT Suite is a collection of standalone Python tools for clearnet intelligence gathering. Each script is self-contained — download only what you need. All use free or freemium data sources, with optional paid API key support for enriched output.
README: https://github.com/N1ckw1ck/Clear_OSINT/blob/main/README.md
Prerequisites
- Python 3.11+
- pip and a virtual environment (recommended)
- Internet access — all tools make outbound requests to public APIs and data sources
Install all dependencies
pip install -r requirements.txt
# All / most tools requests phonenumbers # c_phone_osint.py serpapi # c_url_osint.py beautifulsoup4 dnspython python-whois # exif_tool.py Pillow piexif
requirements.txt. Each tool's install section lists exactly what it needs.
Phone OSINT
Phone number intelligence tool. Validates and parses numbers using the phonenumbers library, checks spam likelihood, geocodes the carrier region to approximate coordinates, and optionally scans the clearnet for mentions of the number in multiple formats.
Setup & API Keys
The tool works without any API keys. Two optional keys will enrich the report with additional data:
# IPQualityScore — fraud score, DNC status, carrier enrichment # Sign up free at ipqualityscore.com IPQS_API_KEY: str = '' # SerpAPI — clearnet mention scan via Google + DuckDuckGo # Sign up free at serpapi.com (100 searches/month) SERPAPI_KEY: str = ''
Install
pip install requests phonenumbers serpapi
Usage
python c_phone_osint.py python c_phone_osint.py --save # auto-save report to JSON
Usage flow
- Select US/Canada or International, then enter the number in prompted parts.
- Main report generates immediately — validation, carrier, region, formats, spam check, geocoordinates, and IPQS enrichment if a key is set.
- Prompted to run a clearnet mention scan (requires SerpAPI key). Default is no.
- If the scan returns more than 10 results, prompted to save all to JSON.
How It Works
- phonenumbers library parses validity, country, region, line type, carrier, and timezone.
- SkipCalls (no key) checks spam likelihood and report count.
- Nominatim / OpenStreetMap (no key) geocodes the carrier region to approximate coordinates — city or state level, not exact.
- IPQualityScore (optional key) adds fraud score, DNC registry status, prepaid flag, recent abuse flag, VOIP flag, MCC/MNC, and CNAM lookup.
- SerpAPI (optional key) searches Google and DuckDuckGo for the number in E.164, national, international, and digits-only formats, deduplicates results, and categorises each hit — scam report, data broker, social media, business directory, or general web.
Notes
- Spam likelihood via SkipCalls produces false negatives, making it a signal, not a definitive verdict.
- Carrier detection is unreliable for US numbers specifically. Paid services do better.
- Geocoding is region/city level, not a precise location.
- IPQS can be troublesome to sign up for. It is advised to do so without a VPN and from a standard browser. Their duplicate account detection is aggressive.
- SerpAPI's free tier allows 100 searches per month. The mention scan consumes 2 per run (one per engine).
Domain OSINT
Comprehensive domain intelligence tool. Resolves DNS records, fetches WHOIS data, parses the SSL certificate, analyses HTTP headers and security posture, fingerprints web technologies, probes sensitive paths, optionally scans for WordPress vulnerabilities, checks email security records, and crawls the site for emails and external domain references.
Install
pip install requests beautifulsoup4 dnspython python-whois
Usage
python c_url_osint.py python c_url_osint.py --save # auto-save report to JSON
Usage flow
- Enter the target domain (with or without
https://) and a max page crawl limit (1–50, default 20). - DNS, WHOIS, SSL, HTTP headers, tech fingerprint, CSP quality, HTTP method enumeration, email security (SPF / DMARC / DKIM), and sensitive path probe run automatically.
- Prompted to run an optional URL threat scan (scrapes IPQS public page).
- Prompted to run an optional port scan against the resolved IP — ensure you are authorized.
- Prompted to run an optional page crawl up to your specified limit.
- Prompted to save the full report to JSON.
How It Works
- Resolves A, PTR, MX, NS, and TXT records via
dnspython. - Fetches WHOIS registration data including registrar, creation and expiry dates, and nameservers.
- Retrieves and parses the SSL/TLS certificate — issuer, expiry, days remaining, Subject Alternative Names.
- Fetches HTTP response headers and checks for the presence or absence of seven key security headers with vulnerability context.
- Probes a list of sensitive paths —
.git/HEAD,.env,phpinfo.php, WordPress endpoints, and more — and reports status codes and content hints. - Evaluates CSP header quality — flags
unsafe-inline,unsafe-eval, wildcard sources, and missing directives. - Fingerprints web technologies from headers and HTML — server, CMS, JS framework, CDN, and hosting provider.
- If WordPress is detected, extracts version from up to four sources and maps it against known CVEs.
- Checks SPF, DMARC, and DKIM selectors via DNS TXT lookups with policy grading.
- Calculates a passive risk score (0–100) from already-collected data — no extra requests.
- Optionally probes 14 ports via direct TCP connect with banner grabbing on text-protocol ports.
- Optionally crawls the domain with a 1-second delay between pages, collecting emails, internal links, and external domain references.
Notes
- The crawl uses a 1-second delay between requests out of politeness. On large sites, the page cap will be reached — raise it carefully.
- Technology fingerprinting is based on headers and static HTML. Client-side-rendered sites may not reveal their full stack this way.
- The URL threat scan scrapes a public page rather than using an API. It may break if the page structure changes.
- WHOIS data availability varies by TLD. Privacy-protected domains return minimal registrant information.
- The risk score is a passive indicator derived from your scan data, not a dedicated threat intelligence feed.
- As of April 2026, the tool can scrape sites behind Cloudflare and similar CDNs for email and link extraction.
IP OSINT
IP address intelligence tool. Covers geolocation, ASN and BGP routing data, passive DNS, DNS chain resolution, hosted domain discovery via reverse IP lookup and TLS certificate SANs, Tor exit node check, host classification, and optional ICMP ping and port scan.
Setup & API Keys
The tool works without any API keys. Two optional keys unlock additional data:
# AbuseIPDB — abuse confidence score, report count, usage type # Free tier available at abuseipdb.com ABUSEIPDB_API_KEY: str = '' # Shodan — open ports, service banners, hostnames, OS, tags # Requires paid membership for /shodan/host/{ip} endpoint SHODAN_API_KEY: str = ''
Install
pip install requests
Usage
python c_ip_osint.py python c_ip_osint.py --save # auto-save report to JSON
Usage flow
- Enter a target IPv4 or IPv6 address.
- Private or loopback IPs trigger a local interface scan and exit.
- For public IPs, geolocation, ASN, routing, passive DNS, DNS chain, domain discovery, Tor check, and host classification run automatically. AbuseIPDB and Shodan enrichment run if keys are set.
- Prompted to run an optional ICMP ping (10 packets).
- Prompted to run an optional common port probe (24 ports, direct TCP connect).
- Prompted to save the full report to JSON.
Test target
The Nmap project maintains a publicly scannable host at 45.33.32.156 (scanme.nmap.org) you can use to test the tool.
How It Works
- Classifies the IP (public/private/loopback, IPv4/IPv6) and performs a reverse DNS lookup.
- Fetches geolocation, timezone, and ASN data from
ipapi.co— 1000 requests/day free, no key required. - Queries RIPEstat for BGP prefix, announcement status, RIR allocation, ASN holder, abuse contacts, and routing peers.
- Fetches passive DNS records from RIPEstat showing domain names historically resolved to the target IP.
- Resolves a recursive DNS chain from the reverse DNS hostname.
- Performs reverse IP domain discovery by querying HackerTarget, extracting TLS certificate SANs, cross-referencing Shodan and passive DNS, then HTTP-probing each discovered domain with a Host header to confirm live services.
- Checks the IP against the Tor Project's bulk exit node list.
- Classifies the host as datacenter/cloud, residential ISP, or mobile carrier based on ASN name, org string, and BGP prefix size.
- Optionally sends ICMP ping via the system
pingbinary and parses RTT and packet loss statistics. - Optionally probes 24 common ports concurrently via direct TCP connect, with banner grabbing on text-protocol ports.
Notes
- HackerTarget reverse IP lookup has a limit of approximately 100 requests per day per IP. The geocoding via
ipapi.coallows 1000 per day. - Geolocation precision varies significantly by IP and provider. ISP-owned blocks may only resolve to country or region level.
- Passive DNS coverage depends on what RIPEstat has observed. Many IPs return no records.
- The port probe uses short TCP connect timeouts. Firewalls that drop rather than reject will cause ports to time out silently — false negatives are possible.
- Shodan's free API tier does not include access to the
/shodan/host/{ip}endpoint. A paid membership is required. The tool works fine without it. - An AbuseIPDB score of 0 does not guarantee an IP is clean. A score of 100 does not confirm it is malicious.
EXIF Tool
Reads, displays, strips, and writes EXIF metadata from image files. Supports JPEG, TIFF, PNG and most Pillow-readable formats. GPS coordinates are decoded to decimal degrees with a Google Maps link. Operates interactively or on paths passed as arguments.
Install
pip install Pillow piexif
Usage
python exif_tool.py # interactive mode python exif_tool.py photo.jpg # single file python exif_tool.py photo1.jpg photo2.jpg # multiple files
Usage flow (per file)
- All EXIF fields are printed — tag name alongside decoded value.
- Prompted to strip EXIF.
ystrips with a.backupcopy.bstrips without backup. Enter skips. - Prompted to add or edit EXIF fields. Choose Merge (keeps existing, overwrites selected fields) or Wipe (clears all fields first then writes).
- Select which fields to set from a numbered menu, enter values, then
dto apply.
Writable fields
| Field | Format |
|---|---|
| DateTime | YYYY-MM-DD HH:MM:SS or YYYY:MM:DD HH:MM:SS |
| ImageDescription | Free text |
| Artist | Free text |
| Copyright | Free text |
| Make | Device manufacturer |
| Model | Device model |
| Software | Free text |
| UserComment | Free text (stored as ASCII) |
| GPS | lat,lon — e.g. 51.5074,-0.1278 |
How It Works
- Opens images with Pillow and reads raw EXIF via
_getexif(). Tag IDs are resolved usingPIL.ExifTags.TAGSandGPSTAGS. - GPS IFD entries are decoded from rational tuples to decimal degrees. A Google Maps link is constructed from the coordinates.
- Stripping works by writing an empty piexif block, or — for formats piexif doesn't support — by copying pixel data into a new image with no EXIF.
- Writing uses piexif to load the existing EXIF dict (or start fresh on wipe), apply updates per field, and save back to the original path.
- A
.backupcopy is made before any destructive operation when requested.
Setup & Dependencies
Virtual Environment
Running in a virtual environment keeps dependencies isolated and avoids conflicts with system Python packages. Recommended for all tools.
# Create a venv inside your project folder python3 -m venv .venv # Activate (macOS / Linux) source .venv/bin/activate # Activate (Windows) .venv\Scripts\activate # Install dependencies pip install -r requirements.txt # Deactivate when done deactivate
API Keys
All tools run without API keys. The following optional keys extend what certain tools can return. All linked services offer a free tier.
| Key | Tool | What it adds | Free tier |
|---|---|---|---|
| IPQS | Phone OSINT | Fraud score, DNC status, CNAM, VOIP/prepaid flags | Yes — ipqualityscore.com |
| SerpAPI | Phone OSINT | Clearnet mention scan via Google + DuckDuckGo | Yes — serpapi.com |
| AbuseIPDB | IP OSINT | Abuse confidence score, report count, usage type | Yes — abuseipdb.com |
| Shodan | IP OSINT | Open ports, service banners, hostnames, OS, tags | No — paid plan required for host lookup |
Keys are pasted directly into the relevant script at the top of the file where indicated. They are never sent anywhere other than the respective API endpoint.
Rate Limits
The tools rely on several free public APIs and data sources. Keep the following limits in mind when running scans repeatedly.
| Source | Used by | Limit |
|---|---|---|
| ipapi.co | IP OSINT | 1,000 requests / day (no key) |
| HackerTarget | IP OSINT | ~100 requests / day per IP |
| Nominatim | Phone OSINT | 1 request / second — do not batch |
| RIPEstat | IP OSINT | Reasonable use, no hard limit |
| SerpAPI | Phone OSINT | 100 searches / month (free tier) |
| AbuseIPDB | IP OSINT | 1,000 checks / day (free tier) |
| SkipCalls | Phone OSINT | Reasonable use, no published limit |
Disclaimer & Use
- These tools are for authorized and ethical use only. You are responsible for ensuring your use complies with all applicable laws, regulations, and terms of service.
- Port scanning an IP you do not own or have explicit permission to scan is illegal in many jurisdictions. The IP OSINT tool prompts before port scanning, make sure you are authorized.
- Crawling or scraping websites may violate their terms of service. The domain tool prompts before crawling.
- Requests made by these tools are potentially traceable back to your IP. If you require anonymity for your research, route through a VPN before running any tool.
- The author does not assume liability for any misuse of these tools.