DNS Over Https Proxy Server Implementation In Python — Complete Documentation & Simulation

Overview & System Context

Dns Over Https Proxy Server Implementation In Python: Comprehensive Web-Sourced Technical Investigation Exhaustive Research Dossier · Pure Web-Scraped Content: ~2753 Words · Real Software Screenshots: 3 1. Executive Overview & Technical Background A proxy auto-config (PAC) file defines how web browsers and other user agents can automatically choose the appropriate proxy server (access method) for fetching a given URL. A PAC file contains a JavaScript function FindProxyForURL(url, host).

This function returns a string with one or more access method specifications. These specifications cause the user agent to use a particular proxy server or to connect directly. Multiple specifications provide a fallback when a proxy fails to respond. The browser fetches this PAC file before requesting other URLs. The URL of the PAC file is either configured manually or determined automatically by the Web Proxy Auto-Discovery Protocol.

Automatic proxy selection: Specify a host-name and a port number to be used for all URLs. Most browsers allow you to specify a list of domains (such as localhost) that will bypass this proxy. Proxy auto-configuration (PAC): Specify the URL for a PAC file with a JavaScript function that determines the appropriate proxy for each URL. This method is more suitable for laptop users who need several different proxy configurations, or complex corporate setups with many different proxies. Web Proxy Auto-Discovery Protocol (WPAD): Let the browser guess the location of the PAC file through DHCP and DNS lookups.

FindProxyForURL(url, host), with two arguments and return value in specific format: * url is the URL of the object * host is the host-name derived from that URL. Syntactically it is the same string as between :// and the first : or / after that. * return "..." is a string of signatures in the following format (see examples below): To use it, a PAC file is published to a HTTP server, and client user agents are instructed to use it, either by entering the URL in the proxy connection settings of the browser or through the use of the WPAD protocol. The URL may also reference a local file as for example: file:///etc/proxy.pac. Even though most clients will process the script regardless of the MIME type returned in the HTTP reply, for the sake of completeness and to maximize compatibility, the HTTP server should be configured to declare the MIME type of this file to be either application/x-ns-proxy-autoconfig or application/x-javascript-config. There is little evidence to favor the use of one MIME type over the other. It would be, however, reasonable to assume that application/x-ns-proxy-autoconfig will be supported in more clients than application/x-javascript-config as it was defined in the original Netscape specification, the latter type coming into use more recently.

Figure 1: GitHub - sudozid/dnsproxy-gui: GUI Frontend for AdguardTeam/dnsproxy This function instructs the browser to retrieve all pages through the proxy on port 8080 of the server proxy.example.com. Should this proxy fail to respond, the browser contacts the Web-site directly, without using a proxy. The latter may fail if firewalls, or other intermediary network devices, reject requests from sources other than the proxy—a common configuration in corporate networks. A more complicated example demonstrates some available JavaScript functions to be used in the FindProxyForURL function: By default, the PROXY keyword means that a proxy corresponding to the protocol of the original request, be it http, https, or ftp, is used. Other supported keyword and proxy types include: SOCKS Use a SOCKS proxy. HTTP, HTTPS Introduced in more recent versions of Firefox. Specifies an HTTP(S) proxy.

SOCKS4, SOCKS5 Introduced in more recent versions of Firefox. Specifies the SOCKS protocol version. The current convention is to fail over to direct connection when a PAC file is unavailable. Shortly after switching between network configurations (e.g. when entering or leaving a VPN), dnsResolve may give outdated results due to DNS caching. For instance, Firefox usually keeps 20 domain entries cached for 60 seconds. This may be configured via the network.dnsCacheEntries and network.dnsCacheExpiration configuration variables. Flushing the system's DNS cache may also help, which can be achieved e.g. in Linux with sudo service dns-clean start or in Windows with ipconfig /flushdns. On Internet Explorer 9, isInNet('localHostName', 'second.ip', '255.255.255.255') returns true and can be used as a workaround.

The myIpAddress function assumes that the device has a single IPv4 address. The results are undefined if the device has more than one IPv4 address or has IPv6 addresses. Figure 2: Review/Test config [https-dns-proxy 2021-11-22-3] - Installing and ... Server Name Indication (SNI) is an extension to Transport Layer Security (TLS) where a client indicates which hostname it is attempting to connect to at the start of the handshaking process. The extension allows a server to present one of multiple possible certificates on the same IP address and port number, allowing for multiple services (including HTTPS websites) to be served by the same IP address without using a single certificate. For HTTPS, it is the conceptual equivalent to HTTP/1.1 virtual hosting. In the original specification, the hostname is not encrypted; the later Encrypted Client Hello specification rectifies this. The SNI extension was specified in 2003 in RFC 3546.

Domain fronting is a technique of replacing the desired host name in SNI with another one hosted by the same server or, more frequently, network of servers known as a content delivery network. When a client uses domain fronting, it replaces the server domain in SNI (unencrypted), but leaves it in the HTTP host header (which is encrypted by TLS) so that server can serve the right content. Domain fronting violates the standard defining SNI itself, so its compatibility is limited (many services check that SNI host matches the HTTP header host and reject connections with domain-fronted SNI as invalid). While domain fronting was used in the past to avoid government censorship, its popularity dwindled because major cloud providers (Google, Amazon's AWS and CloudFront) explicitly prohibit it in their TOS and have technical restrictions against it.

2. System Architecture & Core Working Principles SOCKS is an Internet protocol that exchanges network packets between a client and server through a proxy server. SOCKS5 optionally provides authentication, so only authorized users may access a server. Practically, a SOCKS server proxies TCP connections to an arbitrary IP address and provides a means for UDP packets to be forwarded. The SOCKS protocol operates between the application layer and the transport layer. A SOCKS server accepts incoming client connection on TCP port 1080.

A circumvention tool, allowing traffic to bypass Internet filtering to access content otherwise blocked, e.g., by governments, workplaces, schools, and country-specific web services. Since SOCKS is very detectable, a common approach is to present a SOCKS interface for more sophisticated protocols: The Tor onion proxy software presents a SOCKS interface to its clients. Providing similar functionality to a virtual private network, allowing connections to be forwarded to a server's "local" network: Some SSH suites, such as OpenSSH, support dynamic port forwarding that allows the user to create a local SOCKS proxy.

This can free the user from the limitations of connecting only to a predefined remote port and server. VER: 8 bits SOCKS version number, 0x04 for this version CMD: 8 bits Command code: 0x01: Establish a TCP/IP stream connection. 0x02: Establish a TCP/IP port binding. DSTPORT: 16 bits Destination port number, in network byte order. DESTIP: 32 bits Destination IPv4 address, in network byte order. ID: variable The User ID string, null-terminated. The server responds with: VN: 8 bits Reply version (a null byte). REP: 8 bits Reply code: 0x5a: Request granted. 0x5b: Request rejected or failed. 0x5c: Request failed because client is not running identd (or not reachable from server). 0x5d: Request failed because client's identd could not confirm the user ID in the request. DSTPORT: 16 bits Destination port number, meaningful if granted in BIND, should otherwise be ignored. DESTIP: 32 bits Destination IPv4 address. Together with DSTPORT the ip:port the client should bind to.

Figure 3: How to Configure DNS over HTTPS (DoH) with DNSCrypt-Proxy on OPNsense ... For example, this is a SOCKS4 request to connect Fred to 66.102.7.99:80, the server replies with an "OK": Client: 0x04 | 0x01 | 0x00 0x50 | 0x42 0x66 0x07 0x63 | 0x46 0x72 0x65 0x64 0x00 The last field is "Fred" in ASCII, followed by a null byte. Server: 0x00 | 0x5A | 0xXX 0xXX | 0xXX 0xXX 0xXX 0xXX 0xXX can be any byte value. The SOCKS4 protocol specifies that the values of these bytes should be ignored. From this point onwards, any data sent from the SOCKS client to the SOCKS server is relayed to 66.102.7.99, and vice versa. The command field may be 0x01 for "connect" or 0x02 for "bind"; the "bind" command allows incoming connections for protocols such as active FTP.

SOCKS4_C SOCKS4 client handshake packet (above) DOMAIN the domain name of the host to contact, null (0x00) terminated Server to SOCKS client: (Same as SOCKS4) A server using protocol SOCKS4a must check the DSTIP in the request packet. If it represents address 0.0.0.x with nonzero x, the server must read in the domain name that the client sends in the packet. The server should resolve the domain name and make connection to the destination host if it can.

Client connects and sends a greeting, which includes a list of authentication methods supported. Server chooses one of the methods (or sends a failure response if none of them are acceptable). Several messages may now pass between the client and the server, depending on the authentication method chosen. Client sends a connection request similar to SOCKS4. Server responds similar to SOCKS4. The initial greeting from the client is: VER SOCKS version (0x05) NAUTH Number of authentication methods supported, uint8 AUTH Authentication methods, 1 byte per method supported The authentication methods supported are numbered as follows: 0x00: No authentication 0x01: GSSAPI (RFC 1961) 0x02: Username/password (RFC 1929) 0x03–0x7F: methods assigned by IANA 0x03: Challenge–Handshake Authentication Protocol 0x04: Unassigned 0x05: Challenge–Response Authentication Method 0x06: Secure Sockets Layer 0x07: NDS Authentication 0x08: Multi-Authentication Framework 0x09: JSON Parameter Block 0x0A–0x7F: Unassigned 0x80–0xFE: methods reserved for private use VER SOCKS version (0x05) CAUTH chosen authentication method, or 0xFF if no acceptable methods were offered The subsequent authentication is method-dependent. Username and password authentication (method 0x02) is described in RFC 1929: 3. Software Implementation, Frameworks & Protocols VER 0x01 for current version of username/password authentication IDLEN, ID Username length, uint8; username as bytestring PWLEN, PW Password length, uint8; password as bytestring VER 0x01 for current version of username/password authentication STATUS 0x00 success, otherwise failure, connection must be closed After authentication the connection can proceed. We first define an address datatype as: TYPE type of the address. One of: 0x01: IPv4 address 0x03: Domain name 0x04: IPv6 address ADDR the address data that follows. Depending on type: 4 bytes for IPv4 address 1 byte of name length followed by 1–255 bytes for the domain name 16 bytes for IPv6 address VER SOCKS version (0x05) CMD command code: 0x01: establish a TCP/IP stream connection 0x02: establish a TCP/IP port binding 0x03: associate a UDP port RSV reserved, must be 0x00 DSTADDR destination address, see the address structure above. DSTPORT port number in a network byte order VER SOCKS version (0x05) STATUS status code: 0x00: request granted 0x01: general failure 0x02: connection not allowed by ruleset 0x03: network unreachable 0x04: host unreachable 0x05: connection refused by destination host 0x06: TTL expired 0x07: command not supported / protocol error 0x08: address type not supported RSV reserved, must be 0x00 BNDADDR server bound address in the "SOCKS5 address" format specified above BNDPORT server bound port number in a network byte order Since clients are allowed to use either resolved addresses or domain names, a convention from cURL exists to label the domain name variant of SOCKS5 "socks5h", and the other simply "socks5". A similar convention exists between SOCKS4a and SOCKS4.

Proxifier, a paid proprietary program for Windows and macOS that can force programs to use a SOCKS, HTTPS or HTTP proxy using the Windows Filtering Platform (on Windows) or the Network Extension Framework (on macOS). tun2socks, an open-source tool that creates virtual TCP TUN adapters from a SOCKS proxy, capable of UDP if supported on another end. Works on Linux and Windows, has a macOS port and reimplementation in Golang. Another implementation, written in C with good performance, works on Linux/Android/BSD/macOS and iOS. proxychains, a Unix program that forces TCP traffic through SOCKS or HTTP proxies on (dynamically linked) programs it launches. Works on various Unix-like systems.

Lack of request and packets exchange encryption makes SOCKS practically vulnerable to man-in-the-middle attacks and IP addresses eavesdropping, which in consequence clears a way to censorship by governments. A set of python 3 scripts that supports proxying DNS over HTTPS as specified in the IETF Draft draft-ietf-doh-dns-over-https . DOH provides a way to run encrypted DNS over HTTPS, a protocol which can freely traverse firewalls when other encrypted mechanism may be blocked.

DOH Proxy was created during IETF Hackathon 100 as a proof-of-concept and is not used at Facebook. 4. Practical Deployment, Analysis & Case Studies doh-proxy is a stand alone server answering DOH request. The proxy does not do DNS recursion itself and rather forward the query to a full-featured DNS recursive server or DNS caching server.

doh-httpproxy is designed to be running behind a reverse proxy. In this setup a reverse proxy such as NGINX would be handling the HTTPS/HTTP2 requests from the DOH clients and will forward them to doh-httpproxy backends. While this setup requires more upfront setup, it allows running DOH proxy unprivileged and on multiple cores. doh-stub is the piece of software that you would run on the clients. By providing a local DNS server, doh-stub will forward the DNS requests it receives to a DOH server using an encrypted link.

doh-client is just a test cli that can be used to quickly send a request to a DOH server and dump the returned answer. From within the root of the repository, you can test the proxy, stub and client respectively by using the following commands: This document will focus on the doh-proxy tools arguments and where they should be run. The specifics of configuring a DNS recursive resolver, reverse proxy are outside the scope of this document and are already intensively covered o the Internet.

In this setup, we will run a reverse proxy server that will take care of handling https request and forward them to a dns-httpproxy that runs on the same host. Assuming we use nginx as our reverse proxy and 2 instances of doh-httpproxy , one listening on port 8080 and the other one on port 8081.

Provides a simple Python based proxy for running DNS over HTTPS to Google's DNS over HTTPS service. 5. Performance Evaluation & Extended Technical Observations Recently I wrote a blog post which probably gives you just enough information to get this up and running on a Mac / Linux box... https://robertputt.co.uk/securing-dns-traffic-with-dns-over-https.html , This script plays nice with both Python2.7 and Python 3.x Provides a simple Python based proxy for running DNS over HTTPS to Google's DNS over HTTPS service.

This script runs a local DNS server that forwards queries securely using DNS-over-HTTPS (DoH) instead of standard DNS. It sends encrypted queries to configurable DoH providers, significantly enhancing online privacy and security by preventing DNS eavesdropping and manipulation. Additionally, it can optionally check requested domains against VirusTotal and/or IsMalicious APIs to block access to known malicious or suspicious sites before resolving them.

Important : Running a server on port 53 typically requires root/administrator privileges. Client Configuration : After starting the server, configure your operating system or specific devices to use the IP address of the machine running SDNS-ProxyServer as their primary DNS server. This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

DNS proxy server written in Python that securely forwards DNS queries using DNS-over-HTTPS. Many options for flexible setup, along with a malicious domain checking and blocking function. DNS-over-HTTPS (DoH, RFC 8484) encrypts DNS queries within HTTPS traffic on port 443. Unlike traditional DNS (plaintext UDP port 53) and DNS-over-TLS (dedicated port 853), DoH hides DNS queries inside normal HTTPS traffic, making them indistinguishable from web browsing. This prevents ISPs from logging DNS queries and blocks DNS manipulation by network intermediaries.

This guide covers a Mermaid comparison of plain DNS vs DoH resolution flows, Nginx DoH server configuration with TLS 1.3, a Python client that sends RFC 8484 DNS queries to any DoH provider, systemd-resolved and Firefox DoH setup, and enterprise deployment architecture with internal DoH servers. Harvested Online Documentation & Reference Sources • https://en.wikipedia.org/wiki/Proxy%20auto-config • https://en.wikipedia.org/wiki/Server%20Name%20Indication • https://en.wikipedia.org/wiki/SOCKS • https://chantra.github.io/doh-proxy/ • https://github.com/robputt/Py-DNS-over-HTTPS-Proxy • https://github.com/MrCyberDuck/SDNS-ProxyServer • https://calmops.com/network/dns-over-https-doh/ • https://github.com/mstephen77/dns-doh-proxy