Tuesday, December 19, 2017

HTTP Status codes & 7 most common HTTP Methods for API Testing

HTTP STATUS CODES

The list of http status codes are listed here and also in RFC 7230. However, the status codes do not define the error response when things go wrong. The error response format is defined in RFC 7807 .
Overall the status codes are in the range of 1xx-5xx & are divided into below 5 classes with first digit signifying the class of response.
  1. 1xx Informational response
  2. 2xx Success
  3. 3xx Redirection
  4. 4xx Client errors
  5. 5xx Server errors
The following codes are not specified by any standard.

103 Checkpoint
Used in the resumable requests proposal to resume aborted PUT or POST requests.
218 This is fine (Apache Web Server)
Used as a catch-all error condition for allowing response bodies to flow through Apache when ProxyErrorOverride is enabled. When ProxyErrorOverride is enabled in Apache, response bodies that contain a status code of 4xx or 5xx are automatically discarded by Apache in favor of a generic response or a custom response specified by the ErrorDocument directive.
419 Page Expired (Laravel Framework)
Used by the Laravel Framework when a CSRF Token is missing or expired.
420 Method Failure (Spring Framework)
A deprecated response used by the Spring Framework when a method has failed.
420 Enhance Your Calm (Twitter)
Returned by version 1 of the Twitter Search and Trends API when the client is being rate limited; versions 1.1 and later use the 429 Too Many Requests response code instead.
430 Request Header Fields Too Large (Shopify)
Used by Shopify, instead of the 429 Too Many Requests response code, when too many URLs are requested within a certain time frame.
450 Blocked by Windows Parental Controls (Microsoft)
The Microsoft extension code indicated when Windows Parental Controls are turned on and are blocking access to the requested webpage.
498 Invalid Token (Esri)
Returned by ArcGIS for Server. Code 498 indicates an expired or otherwise invalid token.
499 Token Required (Esri)
Returned by ArcGIS for Server. Code 499 indicates that a token is required but was not submitted.
509 Bandwidth Limit Exceeded (Apache Web Server/cPanel)
The server has exceeded the bandwidth specified by the server administrator; this is often used by shared hosting providers to limit the bandwidth of customers.[79]
526 Invalid SSL Certificate
Used by Cloudflare and Cloud Foundry's gorouter to indicate failure to validate the SSL/TLS certificate that the origin server presented.
529 Site is overloaded
Used by Qualys in the SSLLabs server testing API to signal that the site can't process the request.
530 Site is frozen
Used by the Pantheon web platform to indicate a site that has been frozen due to inactivity.
598 (Informal convention) Network read timeout error
Used by some HTTP proxies to signal a network read timeout behind the proxy to a client in front of the proxy.

Internet Information Services
  1. 440 Login Time-out
  2. 449 Retry With
  3. 451 Redirect
nginx
  1. 444 No Response
  2. 494 Request header too large
  3. 495 SSL Certificate Error
  4. 496 SSL Certificate Required
  5. 497 HTTP Request Sent to HTTPS Port
  6. 499 Client Closed Request
Cloudflare
  1. 520 Web Server Returned an Unknown Error
  2. 521 Web Server Is Down
  3. 522 Connection Timed Out
  4. 523 Origin Is Unreachable
  5. 524 A Timeout Occurred
  6. 525 SSL Handshake Failed
  7. 526 Invalid SSL Certificate
  8. 527 Railgun Error
  9. 530
AWS Elastic Load Balancer
  1. 460 - Client closed the connection with the load balancer before the idle timeout period elapsed. Typically when client timeout is sooner than the Elastic Load Balancer's timeout.
  2. 463 - The load balancer received an X-Forwarded-For request header with more than 30 IP addresses.

HTTP Methods

  1. GET
  2. POST
  3. PUT
  4. DELETE
  5. HEAD
  6. PATCH
  7. OPTIONS
GET requests are used to retrieve data from an API server at the specified resource. A successful/valid GET request will return a response with status code 200.

POST requests are used to send data from an API server to create or update a resource. A successful/valid POST request will return a response with status code 201.

PUT requests are used to send data to the API to create or update a resource. The difference is that PUT requests are idempotent i.e. calling the same PUT request multiple times will always produce the same result
In contrast, calling a POST request repeatedly may have side effects of creating the same resource multiple times. A successful/valid POST request will return a response with status code 200.

DELETE request deletes the resource at the specified URL. A successful/valid DELETE request will return a response with status code 200. A DELETE request where the specified resource is already deleted or non-existent will return a status code of 404 and a DELETE request to an unknown resource/specified URL will return a non-200 status code.

HEAD requests are same as GET but don't return the response body. For e.g if GET requests returns a list of names in the response body then HEAD request will make the same request but won't return the list of names. HEAD request is generally useful in confirming if a GET request will attempt to retrieve data successfully like before downloading a large file or a response body.

It's important to note that not every endpoint that support GET will support HEAD. This completely depends on the API you're testing.

PATCH applies only partial modifications to the resource. The difference between PATCH and PUT, is that a PATCH request is non-idempotent (like a POST request).
With a PATCH request, you may only need to send the updated username in the request body - as opposed to POST and PUT which require the full user entity.

OPTIONS request returns data describing what other methods & operations the server supports at the given URL. OPTIONS requests are more loosely defined & used than the others, making them a good candidate to test for fatal API errors. If an API isn't expecting an OPTIONS request, it's good to put a test case in place that verifies failing behavior. 


Monday, December 18, 2017

Overlay Networks

Overlay Networks

Network Overlays


These overlays have been in use to extend physical network (underlay) subnets/segments across physical boundaries. The routers and switches work as end points. Some examples are:- 
  1. OTV
  2. LISP
  3. Traditional VPNs
  4. FabricPath (Source Learning) - All Links Active Active

Host Overlays

The Hypervisor Vswitches or Physical switches like N9k act as end points, usually a single domain admin using a single controller like APIC can deploy and administer this kind of network,

VXLAN (Multicast - Flooding - UDP)
NVGRE (Unicast)
STT (Stateless Transport Tunneling)


Hybrid Overlays

The traffic can move between virtual and physical node, both working as end points to have a seamless extension of L2 physical boundary.

VXLAN - A tunneling protocol which encapsulates L2 Ethernet frames in Layer 3 UDP packets on port 4789, this allows L2 subnets spanning across physical L3 networks.

VXLAN has 24 bit = 16 million identifiers whereas VLAN as only 4096


  1. L2 VNI - VXLAN Network Identifier carried in VXLAN packets bridged across VTEPs.
  2. L3 VNI - VXLAN Network Identifier carried in VXLAN packets routed across VTEPs. This VNI is linked per tenant VRFs.
  3. VNI - VXLAN Network Identifier.
  4. VTEP - VXLAN Tunnel Endpoint.
  5. VXLAN L2 Gateway - VTEP capable of switching VLAN-VXLAN, VXLAN-VLAN packets within same VNI.
  6. VXLAN L3 Gateway - VTEP capable of routing VXLAN across different VNIs.

VXLAN packets received from the undelay networks with L4-L3-L2 headers are encapsulated in a new IP + MAC header (overlay) before being multicasted to multicast IP such as 239.1.1.2

Reference :- https://www.youtube.com/watch?v=kAoa7djX3Ew


Wednesday, August 2, 2017

7 Steps for Setting up a CentOS 7 NFS Server from Scratch

Steps:-

1. Setup Network & Hostname
2. Download all required packages (nfs-utils & nfs-utils-lib)
3. Create & Format the Partition (use parted for >2TB shares)
4. Configure the share point, mount the partition & configure auto mount
5. Append /etc/exports & /etc/fstab
6. Configure Firewall to allow NFS service
7. Restart Services & Reboot the Server.


I used the below commands (parted) to setup a 5TB NFS share on CentOS 7 Kernel 3.10.0-514.el7.x86_64

Configure Network & Hostname
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=newHostName
vi /etc/hosts 
127.0.0.1 newHostName
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
and then rebooting the system

Download all packages for NFS
Download all required packages
yum update
yum -y install nfs-utils
yum -y install nfs-utils-lib

yum install  0 0

With Fdisk
Fdisk used to create a new partition on new device sda/sdb
Format the new xfs file system using
# mkfs.xfs /dev/sdb1
Mount the xfs file system
# mkdir /mnt/db
# mount /dev/sdb1 /mnt/db
# mount | grep /dev/sdb1

Create the GPT partition table 
mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) unit GB


With Parted
Check partition using fdisk -l
Run parted utility
Select /dev/sdb

Create the partition
mkpart primary 0.0GB 4000.8GB
Quit parted now

Create the filesystem
mkfs.ext4 /dev/sdb1


Create Share point
mkdir /home/share
Change the permissions
chmod -R 777 /home/share/
Mount the partition
mount /dev/sdb1 /home/share

Automounting NFS Shares on Server Reboot:
Append text to the end of /usr/lib/systemd/system/nfs-idmap.service
[Install]
WantedBy=multi-user.target

Append text to the end of /usr/lib/systemd/system/nfs-lock.service
[Install]
WantedBy=nfs.target


Add the partition to /etc/fstab and mounting it
/dev/sdb1              /home/share        ext4    defaults        0 0


[root@localhost ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Jul  5 17:25:55 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/cl-root     /                       xfs     defaults        0 0
UUID=f2c3a44a-621a-4dd9-b86d-cf5f2fdee4a0 /boot                   xfs     defaults        0 0
/dev/mapper/cl-home     /home                   xfs     defaults        0 0
/dev/mapper/cl-swap     swap                    swap    defaults        0 0
/dev/sdb1 /home/share ext4 defaults 0 0
[root@localhost ~]#


Append /etc/exports
vi /etc/exports
/home/share "192.168.0.0/16"(rw,sync,no_subtree_check,no_root_squash)


Configure Firewall to allow NFS service
firewall-cmd --permanent --zone public --add-service mountd
firewall-cmd --permanent --zone public --add-service rpc-bind
firewall-cmd --permanent --zone public --add-service nfs
firewall-cmd --reload

Restart Services
systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap

systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Reboot the host
Shutdown -r now

Sunday, March 26, 2017

CIFS and SMB


CIFS

Short for Common Internet File System, a protocol that defines astandard for remote file access using millions of computers at a time. CIFS is a dialect of the Server Message Block (SMB) protocol, which was originally developed by IBM Corporation and then further enhanced by Microsoft, IBM, Intel, 3Com, and others

With CIFS, users with different platforms and computers can share files without having to install new software. In general, CIFS is a better option over older file sharing protocols like FTP unless links are high latency (e.g WAN)
CIFS runs over TCP/IP but uses the SMB (Server Message Block)protocol found in Microsoft Windows for file and printer access; therefore, CIFS will allow all applications, not just Web browsers, to open and share files across the Internet.
With CIFS, changes made to a file are simultaneously saved on both the client and server side.

SMB

SMB (v1.0, 2.0 and 3.0) prevalent in newer windows OS versions whereas CIFS is a dialect which doesn't have all the features of SMB such as "batching" or change notification support. As per Microsoft, the protocol exchange between a server and client first selects a dialect with the highest level of functionality supported by both server and client. Initially CIFS or SMB might be selected based on Server & Client capability but if either the Server or Client fail to support all needed features of SMB like change notification support then SMB connections will be lost without notice to user with traffic shifting to CIFS only.


CIFS vs SMB, which one to use?

CIFS being a dialect & older implementation of SMB lacks important enhancements like multi-channeling and change notification support which are present in SMB 2.0 and SMB 3 version. Microsoft has been using SMB from Windows 2008 onwards and it remains the protocol of choice for transferring files between compatible nodes.

References : - 
https://docs.microsoft.com/en-us/windows/desktop/FileIO/microsoft-smb-protocol-and-cifs-protocol-overview
https://msdn.microsoft.com/en-us/library/ee442092.aspx


Windows 2012/2016 Hyper-V Cluster

Setting up a 21012 Hyper-V Failover Clustering

  1. Setup the networks & storage for nodes
  2. Install Hyper-V & Failover Clustering roles on nodes
  3. Validate the cluster configuration 
  4. Create the cluster
  5. Setup disk or file share as witness if required (for even node clusters)
  6. Create a VM & configure it highly available 
  7. Install the guest OS on VM

Witness Configuration

For a two-node cluster, the quorum configuration will be Node and Disk Majoritya the default for a cluster with an even number of nodes. Node and Disk Majority means that the nodes and the witness disk each contain copies of the cluster configuration, and the cluster has quorum as long as a majority (two out of three) of these copies are available.

Troubleshooting Checklist.

  1. For cloned VMs which are going to be part of a domain, ensure the image is sysprep'ed before deploying the VMs.
  2. Network used for clustered are up and named similarly.
  3. The storage disks are up and available, the witness should have ntfs partition and so does the CSV.
  4. If VM migration fails, check no CD/DVD/image is mounted.
  5. Ensure the cluster share has appropriate rights (CSV conversion of a disk does this automatically) in case of using a file share for CSV, reset the folder permission to "Everyone" with full permission. This is assuming that file share is a separate network and only accessible from the HyperV hosts.


Reference :- Windows 2012 HyperV Cluster