A protocol is a set of rules and conventions that govern how data is transmitted and received over a network. Every protocol defines its syntax (structure of data), semantics (meaning of the data), timing (when to send), and expected behavior (how to respond to different situations). Security vulnerabilities often arise when an implementation assumes the other party follows the protocol correctly — but attackers deliberately deviate from it.

​
TCP vs. UDP

Both TCP and UDP operate at the Transport Layer (Layer 4) of the OSI model. They serve different purposes depending on whether reliability or speed matters more.
Transmission Control Protocol (TCP) is connection-oriented. Before any data moves, both sides complete a three-way handshake:
  1. SYN — Client sends a synchronize packet to initiate the connection.
  2. SYN-ACK — Server acknowledges and signals its willingness to connect.
  3. ACK — Client confirms, completing the handshake.
TCP then guarantees:
  • Delivery of all packets
  • In-order reassembly
  • Flow control (don’t overwhelm the receiver)
  • Congestion control
Common uses: Web browsing (HTTP/HTTPS), email (SMTP, IMAP), file transfer (FTP).Security risk — TCP SYN Flood: An attacker sends a large number of SYN packets but never sends the final ACK. The server allocates resources for each half-open connection. If enough of these pile up, the server runs out of resources and cannot accept legitimate connections — a denial-of-service (DoS) attack.

​
DNS — Domain Name System

DNS translates human-readable domain names (e.g., www.example.com) into IP addresses (e.g., 192.0.2.1). Without DNS, you would need to memorize IP addresses for every service you use. When your browser navigates to a URL, it sends a DNS query (over UDP, port 53) to a resolver. The resolver walks a hierarchy of DNS servers until it finds the authoritative answer and returns it to your device. DNS supports several record types:
RecordPurpose
AMaps a domain name to an IPv4 address
AAAAMaps a domain name to an IPv6 address
MXSpecifies the mail server for a domain
CNAMECreates an alias pointing to another domain name
Security risks:
  • DNS cache poisoning: An attacker inserts a forged DNS record into a resolver’s cache. Users who query that resolver get directed to a malicious IP address instead of the legitimate one.
  • DNS tunneling: Attackers encode data inside DNS queries and responses to exfiltrate information through firewalls that allow DNS traffic.
  • Amplification attacks: Small DNS queries generate large responses, which can be directed at a victim via spoofed source IPs.
  • Typosquatting: Registering domains that closely resemble legitimate ones (e.g., paypa1.com) to trick users.
DNSSEC (DNS Security Extensions) adds cryptographic signatures to DNS records, allowing resolvers to verify that responses have not been tampered with. It does not encrypt DNS traffic — for that, use DNS over HTTPS (DoH) or DNS over TLS (DoT).

​
DHCP — Dynamic Host Configuration Protocol

When your device connects to a network, it needs an IP address, subnet mask, default gateway, and DNS server address. DHCP automates this assignment so network administrators don’t have to configure each device manually. The exchange follows four steps (DORA):
  1. DHCPDISCOVER — Your device broadcasts to find a DHCP server.
  2. DHCPOFFER — The server responds with an available IP address and configuration.
  3. DHCPREQUEST — Your device requests the offered address.
  4. DHCPACK — The server confirms the lease.
DHCP uses UDP (ports 67 for server, 68 for client). Leases expire after a set period; your device must renew them to keep the same address. Security risks:
  • Rogue DHCP server: An attacker runs an unauthorized DHCP server on the network. Devices that receive its offers get incorrect IP addresses or — more dangerously — a malicious DNS server address, redirecting all their DNS queries through the attacker.
  • DHCP starvation: An attacker floods the network with DHCPDISCOVER requests to exhaust the pool of available IP addresses, preventing legitimate devices from getting addresses.
DHCP traffic is cleartext. Use DHCP snooping on managed switches to restrict which ports are allowed to respond to DHCP requests, preventing rogue DHCP servers from operating on your network.

​
ARP — Address Resolution Protocol

ARP resolves the gap between IP addresses (Layer 3) and MAC addresses (Layer 2). When a device wants to send data to an IP address on the same local network, it broadcasts an ARP request: “Who has IP address X? Tell me your MAC address.” The device with that IP responds with its MAC address. Your device stores these mappings in an ARP cache to avoid repeated broadcasts. You can inspect it with arp -n (Linux/macOS) or Get-NetNeighbor (Windows). Security risk — ARP poisoning (ARP spoofing): ARP has no authentication. An attacker can send unsolicited ARP replies that associate their MAC address with the IP address of another device (e.g., the default gateway). Once poisoned, all traffic intended for the gateway flows through the attacker first — a classic man-in-the-middle (MitM) attack.
If you see multiple ARP replies for a single IP address, you may be under an ARP spoofing attack. Normally, only one device should respond to an ARP request.

​
TLS — Transport Layer Security

TLS is a cryptographic protocol that provides a secure channel over an otherwise insecure connection. It is the foundation of HTTPS, SMTPS, FTPS, and many other secure variants of existing protocols. TLS provides three core guarantees:
  • Confidentiality: Data is encrypted so eavesdroppers cannot read it (using symmetric encryption like AES after key exchange).
  • Integrity: Message authentication codes (MACs) ensure data has not been modified in transit.
  • Authentication: Digital certificates issued by trusted Certificate Authorities (CAs) verify the server’s identity, preventing impersonation.

​
The TLS handshake

Before data flows, TLS negotiates a shared session key using asymmetric cryptography, then switches to faster symmetric encryption for the actual data transfer.
1

Client Hello

Your browser sends its supported TLS versions, cipher suites, and a random nonce to the server.
2

Server Hello

The server responds with its chosen TLS version and cipher suite, its digital certificate, and its own random nonce.
3

Key exchange

Client and server perform a key exchange (typically Diffie-Hellman) to derive a shared session key without transmitting it directly over the network.
4

Finished

Both parties send a Finished message encrypted with the session key to confirm the handshake succeeded.
5

Secure communication

All subsequent data is encrypted with the agreed session key using symmetric encryption (e.g., AES).
TLS is a successor to SSL (Secure Sockets Layer). If you encounter references to SSL in older documentation, assume TLS is now used instead.

​
HTTP and HTTPS

Both HTTP and HTTPS operate at the Application Layer (Layer 7) and follow a request-response model. Your browser sends a request (e.g., GET /index.html) and the server sends back a response.
HTTPHTTPS
Port80443
EncryptionNone (cleartext)TLS-encrypted
IntegrityNoneTLS MAC
AuthenticationNoneServer certificate
RiskAll traffic readable by anyone on the network pathResistant to eavesdropping and MitM (if certificate is valid)
HTTP is stateless — each request is independent. Cookies and sessions are layered on top to maintain state between requests. Security risks with HTTP:
  • Anyone on the same Wi-Fi or network path can read all traffic, including passwords, cookies, and personal data (packet sniffing).
Security risks with HTTPS:
  • If an attacker can obtain or forge a valid certificate for a domain, they can intercept HTTPS traffic. This is why certificate transparency logs and strict CA practices matter.
  • In corporate environments, some firewalls perform TLS/SSL inspection — they terminate your TLS connection and re-encrypt it with their own certificate, allowing them to inspect the plaintext. Your browser trusts this because the corporate CA is in its trust store.

​
Other application layer protocols

FTP transfers files between a client and server. It uses two channels: a control channel (port 21) for commands and a data channel (port 20) for file transfers. FTP operates in either active mode (server connects back to client) or passive mode (client connects to server on a random port).Critical flaw: FTP transmits all data — including usernames and passwords — in cleartext. Always use SFTP (SSH File Transfer Protocol) or FTPS (FTP over TLS) instead of plain FTP.
SMTP sends email between mail servers and from mail clients to servers. It uses port 25 for server-to-server communication and port 587 for client submission. Secure variants use TLS (port 465 or STARTTLS on 587).Critical flaw: SMTP trusts the sender address in the email header, making email spoofing trivial — an attacker can send an email that appears to come from any address. Protocols like SPF, DKIM, and DMARC provide authentication to mitigate this, but adoption is incomplete.
Many older protocols (HTTP, SMTP, FTP) are text-based and can be interacted with directly using telnet or netcat (nc). This is useful for learning and debugging, but it also means attackers can craft raw protocol messages without needing a dedicated client.