SQL CREATE LOGIN

The CREATE LOGIN Syntax Breakdown

The T-SQL syntax for CREATE LOGIN supports multiple credential sources. Below is the standard syntax structure used across production environments:

SQL

-- Creating a native SQL Server Authentication Login
CREATE LOGIN login_name 
WITH PASSWORD = 'strong_password' 
    [ MUST_CHANGE ]
    [ , DEFAULT_DATABASE = database_name ]
    [ , DEFAULT_LANGUAGE = language_name ]
    [ , CHECK_EXPIRATION = { ON | OFF } ]
    [ , CHECK_POLICY = { ON | OFF } ]
    [ , CREDENTIAL = credential_name ]
    [ , SID = sid_value ];

-- Creating a Windows-Authenticated Login (Domain or Local)
CREATE LOGIN [DOMAIN\account_name] 
FROM WINDOWS 
    [ WITH DEFAULT_DATABASE = database_name ]
    [ , DEFAULT_LANGUAGE = language_name ];

Constructing Native SQL Server Logins

When building native SQL Server logins, you must enforce policy flags to ensure compliance with enterprise frameworks such as NIST, HIPAA, and SOX.

Basic SQL Authentication Login

To create a standard application login tied to a default transactional database:

SQL

CREATE LOGIN AppSvcUser
WITH PASSWORD = 'P@ssw0rd!Secure#2026$Key',
     DEFAULT_DATABASE = AzureLessons,
     DEFAULT_LANGUAGE = us_english,
     CHECK_EXPIRATION = ON,
     CHECK_POLICY = ON;

Critical Security Parameters Explained

• CHECK_POLICY = ON: Instructs SQL Server to enforce the Windows Server host password complexity rules. This prevents users from selecting trivial or easily brute-forced passwords.
• CHECK_EXPIRATION = ON: Forces the account to respect operating system domain password aging and lifetime limits.
• MUST_CHANGE: Requires the user or application engineer to supply a replacement password on their initial connection. (Note: MUST_CHANGE requires CHECK_EXPIRATION = ON).

After running the query above, I got the expected output shown in the screenshot below.

SQL

-- Forcing immediate password rotation upon first logon
CREATE LOGIN DavidMiller
WITH PASSWORD = 'InitialTempPassword#987',
     MUST_CHANGE,
     DEFAULT_DATABASE = OperationsDB,
     CHECK_EXPIRATION = ON,
     CHECK_POLICY = ON;

Integrating Windows and Active Directory Logins

Windows Authentication reduces operational overhead by centralizing credential lifecycles within Active Directory. When an employee leaves the company, revoking their Active Directory profile instantly revokes their access across all SQL Server instances.

Provisioning an Individual Active Directory User

To grant a corporate domain engineer access to your SQL Server instance:

SQL

CREATE LOGIN [CORP\SarahJenkins]
FROM WINDOWS
WITH DEFAULT_DATABASE = CustomerPortal,
     DEFAULT_LANGUAGE = us_english;

Provisioning an Active Directory Security Group (Best Practice)

Managing permissions on an individual basis creates management drift. In high-scale architectures, always map access to an Active Directory Security Group rather than individual employee accounts:

SQL

CREATE LOGIN [CORP\DataAnalytics_Tier2_Engineers]
FROM WINDOWS
WITH DEFAULT_DATABASE = EnterpriseReporting;

When new engineers join the corporate team, group membership management in Active Directory handles database authentication automatically without requiring explicit DBA intervention on the SQL Server instance.

End-to-End Workflow: From Login Creation to Data Access

A common oversight is creating a login and expecting queries to succeed immediately. Let us step through the sequential administrative pipeline required to grant a login read-only access to a database table.

Administrative Execution Sequence:
[1. CREATE LOGIN] ---> [2. USE Database] ---> [3. CREATE USER] ---> [4. GRANT Permissions]

Step 1: Initialize the Server Login

Switch to the instance context and execute the login creation:

SQL

USE master;
GO

CREATE LOGIN MichaelChang
WITH PASSWORD = 'K9#vX!89mQ@pZ7$wL2#e',
     DEFAULT_DATABASE = FinancialRecords,
     CHECK_EXPIRATION = ON,
     CHECK_POLICY = ON;
GO

Step 2: Provision the Database User

Navigate into the target database container and bind a database user principal to the server-level login:

SQL

USE FinancialRecords;
GO

CREATE USER MichaelChang
FOR LOGIN MichaelChang
WITH DEFAULT_SCHEMA = dbo;
GO

Step 3: Assign Database Roles or Explicit Object Permissions

Apply the principle of least privilege by granting only the required access tiers:

SQL

USE FinancialRecords;
GO

-- Option A: Add the user to a standard built-in fixed role
ALTER ROLE db_datareader ADD MEMBER MichaelChang;

-- Option B: Grant explicit, granular execution rights on a specific object
GRANT SELECT ON dbo.MonthlyLedger TO MichaelChang;
GRANT EXECUTE ON dbo.usp_GenerateQuarterlyStatement TO MichaelChang;
GO

Administrative Maintenance: Altering, Disabling, and Dropping Logins

Database systems require continuous maintenance as staffing and application requirements change.

Modifying Passwords and Unlocking Accounts

To rotate a password or unlock an account locked out due to failed attempts:

SQL

-- Rotate password without forcing a reset
ALTER LOGIN AppSvcUser 
WITH PASSWORD = 'NewValidatedPassword!2026#Alpha';

-- Unlock an account locked by Windows Password Policy
ALTER LOGIN DavidMiller 
WITH UNLOCK;

Disabling vs. Dropping Logins

When an account is suspected of compromise or an application is decommissioned, immediately disable the login rather than dropping it. Disabling prevents authentication while preserving server-level metadata, role memberships, and audit trails:

SQL

-- Disable access immediately
ALTER LOGIN MichaelChang DISABLE;

-- Re-enable when clearance is confirmed
ALTER LOGIN MichaelChang ENABLE;

If you must permanently delete a login, ensure it does not own database schemas or server roles:

SQL

USE master;
GO

-- Drop the server login
DROP LOGIN MichaelChang;
GO

Warning: Dropping a login does not automatically delete the corresponding sys.database_principals entry inside user databases. This creates an Orphaned User. Always clean up the database-level user before or immediately after dropping a server login.

Enterprise Best Practices & Security Hardening

To maintain compliance and protect your database infrastructure, adhere to these enterprise security standards:

• Enforce the Principle of Least Privilege: Never add standard user logins to the sysadmin fixed server role. Limit sysadmin privileges to dedicated, audited emergency administrator accounts.
• Avoid Renaming the sa Account Without Strategy: While disabling the native sa (System Administrator) account and renaming it is a common compliance check, remember that renaming alone does not stop targeted attacks if elevated permissions are broadly distributed.
• Audit Failed Logins Regularly: Configure SQL Server Audit or examine the SQL Server Error Logs for Event ID 18456 (Failed Login Attempt). A high volume of these entries indicates potential brute-force attacks or misconfigured connection strings.
• Isolate Service Accounts: Dedicated application services should each have their own login. Never share a single service account across multiple microservices or reporting tools.
• Automate Schema Validation: Ensure all production databases use explicit schemas (e.g., sales, hr, finance) rather than dumping all database users into the dbo default schema.

Summary

Mastering the CREATE LOGIN command allows you to establish a secure, well-architected perimeter around your SQL Server instances. By separating server-level authentication from database-level authorization, enforcing strict operating system password policies, and prioritizing Active Directory integration, you build a resilient foundation for your enterprise data operations.

You may also like the following articles:

• SQL Server Last Login Date for User
• Error 40 Could Not Open Connection to SQL Server
• How to find SQL Server instance name in SSMS

SQL Normalization

What is SQL Normalization?

SQL normalization is the systematic process of organizing data within a relational database to achieve two primary objectives:

1. Eliminate redundant data: Storing the same piece of information in multiple places wastes storage and increases cache misses.
2. Ensure logical data dependencies: Ensuring that related data items are stored together and dependent values rely strictly on valid primary and candidate keys.

Normalization was first introduced by Edgar F. Codd in 1970 as part of his relational model. It relies on decomposing large, unnormalized tables into smaller, highly focused, and related tables using foreign keys.

The Cost of Poor Schema Design: Data Anomalies

When a schema is poorly structured or left unnormalized, the database becomes vulnerable to three destructive operational flaws known as data anomalies.

1. Insertion Anomaly

An insertion anomaly occurs when you cannot record a piece of data without unnecessarily inserting unrelated information. For instance, if an employee’s department data is stored directly in the employee record, you cannot record a newly created department until at least one employee is hired into it.

2. Update Anomaly

An update anomaly happens when data redundancy forces an application to modify multiple records to reflect a single conceptual change. If an enterprise has 500 records referencing a manager located in Chicago, updating that manager’s office location requires 500 row updates. If even one record fails during the transaction, the database enters an inconsistent state.

3. Deletion Anomaly

A deletion anomaly represents the accidental loss of unintended data when deleting a record. For instance, if a department’s metadata only exists inside the records of the staff assigned to it, removing the last employee in that department permanently erases all institutional knowledge of the department itself.

Core Relational Concepts You Must Understand

Before diving into the normal forms, let’s review the technical terms that govern the mathematical decomposition of relational tables:

• Entity: A distinct real-world object or concept (e.g., Customer, Product, Invoice) represented as a relation/table.
• Attribute: A property or characteristic of an entity (represented as a column).
• Tuple: A single record or row within a table.
• Candidate Key: A minimal set of attributes that uniquely identifies a tuple within a relation.
• Primary Key: The chosen candidate key selected by the database architect to serve as the unique identifier for all tuples in the table.
• Foreign Key: An attribute or collection of attributes in one table that references the primary key of another table.
• Functional Dependency ($X \rightarrow Y$): A constraint between two sets of attributes such that the value of attribute set $X$ uniquely determines the value of attribute set $Y$.
• Composite Key: A primary or candidate key composed of two or more attributes.

First Normal Form (1NF): Atomicity and Uniqueness

A table is in First Normal Form (1NF) if and only if:

1. Every column contains only atomic (indivisible) values.
2. There are no repeating groups or comma-delimited arrays within a single attribute.
3. Each record is uniquely identifiable via a defined primary key.
4. The order in which data is stored does not matter.

Unnormalized Representation (Violates 1NF)

Consider an enterprise managing employee skills where multiple values are bundled together:

Why this violates 1NF: The SkillsAcquired column holds non-atomic lists, and Location combines city and state. Querying for all employees who know Python requires wildcards (LIKE '%Python%'), bypassing indexes and degrading performance.

1NF Schema Transformation

To normalize to 1NF, we decompose non-atomic values into distinct rows and assign a proper composite primary key ((EmployeeID, Skill)):

Now, each column is atomic, and every tuple is addressable. However, we have introduced significant redundancy in employee names and locations.

Second Normal Form (2NF): Eliminating Partial Dependencies

A relation is in Second Normal Form (2NF) if:

1. It meets all criteria of 1NF.
2. It contains no partial dependencies—every non-prime attribute must be fully functionally dependent on the entire primary key, not just a subset of a composite key.

Note: If a 1NF table has a single-attribute primary key, it is automatically in 2NF. 2NF issues only arise when composite primary keys are used.

The 1NF Dependency Problem

In the previous 1NF table, our composite primary key is (EmployeeID, Skill):

• (EmployeeID, Skill) -> FirstName, LastName, City, StateCode (Full Key Dependency)
• EmployeeID -> FirstName, LastName, City, StateCode (Partial Dependency)

The employee’s demographic data depends solely on EmployeeID, not on what skill they hold.

2NF Schema Transformation

We eliminate the partial dependency by decomposing the single 1NF table into two distinct entities:

Table 1: Employees (Primary Key: EmployeeID)

Table 2: EmployeeSkills (Composite Primary Key: EmployeeID, Skill)

Now, updating Sarah Miller’s name or city requires modifying exactly one tuple in the Employees table.

Third Normal Form (3NF): Eliminating Transitive Dependencies

A relation is in Third Normal Form (3NF) if:

1. It meets all criteria of 2NF.
2. It contains no transitive functional dependencies—non-prime attributes must not depend on other non-prime attributes.

Mathematically, if $X \rightarrow Y$ and $Y \rightarrow Z$, then $X \rightarrow Z$ is a transitive dependency. To satisfy 3NF, $Z$ must be decoupled and placed into a separate relation where $Y$ is the primary key.

The 2NF Dependency Problem

Let’s examine an expanded Employees table:

Here, the primary key is EmployeeID. Let’s map the functional dependencies:

• EmployeeID -> DepartmentCode
• DepartmentCode -> DepartmentName, OfficeFloor
• Therefore, EmployeeID -> DepartmentName, OfficeFloor is a transitive dependency.

If David Wilson leaves the company and record 104 is deleted, all knowledge that the Marketing department is on Floor 2 vanishes (deletion anomaly).

3NF Schema Transformation

We decompose the relation to isolate the transitive attributes into their own domain entity:

Table 1: Employees (Primary Key: EmployeeID, Foreign Key: DepartmentCode)

Table 2: Departments (Primary Key: DepartmentCode)

Architectural Rule of Thumb: In the words of Bill Kent, every non-key attribute must provide a fact about “the key, the whole key, and nothing but the key, so help me Codd.”

Boyce-Codd Normal Form (BCNF / 3.5NF)

Boyce-Codd Normal Form is a stricter version of 3NF. A relation is in BCNF if and only if:

• For every functional dependency $X \rightarrow Y$, the determinant $X$ is a superkey or candidate key.

A table can satisfy 3NF but violate BCNF when it has:

• Multiple overlapping candidate keys.
• Candidate keys composed of multiple attributes.
• An attribute in one candidate key that depends on a non-key attribute.

BCNF Anomaly Breakdown

Consider an academic advising registry where:

• Each student can have multiple advisors.
• Each advisor specializes in exactly one academic department.
• For a given department, a student is assigned only one advisor.

Candidate Keys for this relation are:

• (StudentID, Department)
• (StudentID, AdvisorName)

Functional Dependencies:

1. (StudentID, Department) -> AdvisorName (Determinant is candidate key $\rightarrow$ Valid 3NF/BCNF)
2. AdvisorName -> Department (Advisor determines department, but AdvisorName by itself is NOT a candidate key!)

Because AdvisorName is a determinant but not a superkey, this relation violates BCNF.

BCNF Schema Transformation

To satisfy BCNF, we split the dependency into two tables:

Table 1: StudentAdvisorAssignments (Composite PK: StudentID, AdvisorID)

Table 2: Advisors (Primary Key: AdvisorID)

Advanced Normal Forms: 4NF and 5NF

While 3NF and BCNF are the industry standard for most enterprise OLTP databases, complex models with independent multi-valued facts require higher levels of normalization.

Fourth Normal Form (4NF)

A relation is in 4NF if:

1. It is in BCNF.
2. It contains no multi-valued dependencies ($X \twoheadrightarrow Y$).

A multi-valued dependency exists when the presence of two or more independent multi-valued attributes for the same determinant forces the table to store all Cartesian product combinations of those values.

• Example: If an engineer (e.g., Brandon Scott) has 3 independent certifications (AWS, Azure, GCP) and manages 3 independent projects (Apollo, Titan, Vulcan), storing them in one table requires $3 \times 3 = 9$ rows.
• 4NF Solution: Split into two independent tables: EmployeeCertifications(EmployeeID, Certification) and EmployeeProjects(EmployeeID, ProjectID).

Fifth Normal Form (5NF / Project-Join Normal Form)

A relation is in 5NF if:

1. It is in 4NF.
2. It cannot be non-loss decomposed into smaller tables without join dependencies.

5NF handles symmetric constraints where data can be split into three or more separate relations and reconstructed without generating spurious tuples.

Best Practices

1. Design in 3NF by Default: Always model your transactional application schemas in 3NF or BCNF during the conceptual and logical phases.
2. Use Surrogate Keys Wisely: Use auto-incrementing integers or UUIDs as primary keys, but do not omit unique constraints on natural candidate keys.
3. Index Foreign Keys: Normalization decomposes data across multiple tables. To prevent performance bottlenecks during JOIN queries, ensure foreign key columns are indexed.
4. Denormalize Consciously, Not Lazily: Never skip normalization out of convenience. Denormalize only after query profiling shows that specific joins are degrading application performance under heavy read traffic.
5. Enforce Referential Integrity: Use database-level FOREIGN KEY ... ON DELETE/UPDATE constraints rather than relying solely on application-layer logic to prevent orphaned records.

Designing an optimal schema requires balancing operational write performance, data integrity, and analytical read patterns. Mastering the mechanics of 1NF through 5NF ensures your data model remains resilient, maintainable, and anomaly-free as your system scales.

You may also like the following articles:

• SQL Indexes
• SQL Server Views
• SQL Scalar Functions
• SQL OVER Clause

SQL NULL

What Is SQL NULL?

In the ANSI SQL standard, NULL is a special marker used to indicate that a data value does not exist in the database.

It is crucial to understand what NULL is not:

• NULL is not an empty string (''). An empty string is known text with a character length of zero.
• NULL is not a numeric zero (0). Zero is a known integer with a defined arithmetic value.
• NULL is not a boolean FALSE. FALSE is a known truth value.
• NULL is not equal to another NULL. Because two missing values are unknown, the database cannot verify that they are identical.

Three-Valued Logic (3VL): True, False, and Unknown

Standard classical logic operates on a binary model: a predicate is either TRUE or FALSE.

Because NULL represents an unknown quantity, SQL implements Three-Valued Logic (3VL). In 3VL, the result of a comparison can be TRUE, FALSE, or UNKNOWN.

When an expression evaluates to UNKNOWN, a standard WHERE clause treats it as non-matching. SQL filter predicates require an expression to evaluate explicitly to TRUE to return a row.

Truth Tables in Three-Valued Logic

To master SQL query execution, you must understand how logical operators (AND, OR, NOT) evaluate UNKNOWN states.

The AND Truth Table

The AND operator returns TRUE only if both operands are TRUE. If either operand is UNKNOWN and the other is TRUE or UNKNOWN, the result is UNKNOWN.

The OR Truth Table

The OR operator returns TRUE if at least one operand is TRUE, regardless of whether the other operand is UNKNOWN.

The NOT Operator

Negating an UNKNOWN value still yields UNKNOWN:

• NOT (TRUE) = FALSE
• NOT (FALSE) = TRUE
• NOT (UNKNOWN) = UNKNOWN

Comparing Values with NULL: IS NULL vs. = NULL

The single most common mistake in SQL query writing is using equality operators (= or !=) to test for NULL.

Why = NULL Always Fails

Consider this query:

SQL

-- INCORRECT: This query will never return any records
SELECT employee_id, first_name, last_name, commission_pct
FROM employees
WHERE commission_pct = NULL;

When the SQL engine evaluates commission_pct = NULL, it asks: “Is an unknown commission equal to an unknown value?” The answer is UNKNOWN. Because WHERE UNKNOWN does not satisfy the filter, zero rows are returned, even if thousands of rows have NULL in that column. Check out the below screenshot for your reference.

The same logic applies to inequality checks: commission_pct != NULL also evaluates to UNKNOWN.

The Correct Approach: IS NULL and IS NOT NULL

SQL provides dedicated comparison predicates designed specifically to test for missing data:

SQL

-- CORRECT: Returns all records where commission_pct is missing
SELECT employee_id, first_name, last_name, commission_pct
FROM employees
WHERE commission_pct IS NULL;

-- CORRECT: Returns all records where commission_pct contains a known value
SELECT employee_id, first_name, last_name, commission_pct
FROM employees
WHERE commission_pct IS NOT NULL;

NULL in Arithmetic Operations and String Concatenation

Whenever NULL enters a mathematical expression or standard string concatenation, the missing data typically propagates throughout the entire expression.

Arithmetic Propagation

Any arithmetic operation (+, -, *, /, %) involving a NULL operand results in NULL:

SQL

SELECT 
    100 + NULL AS addition_result,       -- Output: NULL
    500 * NULL AS multiplication_result, -- Output: NULL
    NULL / 10  AS division_result;       -- Output: NULL

If an employee’s base salary is $85,000 and their bonus column contains NULL, running base_salary + bonus evaluates to NULL, completely wiping out the base salary calculation unless explicitly handled.

String Concatenation Behavior

In the ANSI SQL standard and engines like PostgreSQL, SQLite, and Oracle, concatenating a string with NULL yields NULL:

SQL

-- Standard SQL concatenation
SELECT first_name || ' ' || middle_name || ' ' || last_name AS full_name
FROM clients;

If middle_name is NULL, the entire full_name column evaluates to NULL.

(Note: Microsoft SQL Server behavior depends on the CONCAT_NULL_YIELDS_NULL setting, but standard practice across all modern engines is to treat NULL concatenation defensively).

SQL Functions for Handling NULL

To prevent calculations and string operations from evaluating to NULL, SQL provides built-in functions to substitute default fallback values.

1. COALESCE(): The ANSI Standard Universal Handler

The COALESCE() function evaluates arguments in sequential order and returns the first non-NULL expression. If all arguments are NULL, it returns NULL.

SQL

SELECT 
    customer_id,
    first_name,
    last_name,
    COALESCE(phone_number, mobile_number, emergency_contact, 'No Phone Provided') AS primary_contact
FROM customers;

COALESCE() is ANSI SQL standard and runs identically across PostgreSQL, MySQL, SQL Server, Oracle, and SQLite.

2. NULLIF(): Preventing Division by Zero

The NULLIF(expr1, expr2) function compares two expressions:

• If expr1 = expr2, it returns NULL.
• If expr1 != expr2, it returns expr1.

The most powerful application of NULLIF() is guarding against runtime division by zero errors:

SQL

-- Prevents a divide-by-zero fatal error by turning 0 into NULL
SELECT 
    product_name,
    total_revenue / NULLIF(units_sold, 0) AS average_price_per_unit
FROM product_sales;

When units_sold is 0, NULLIF(units_sold, 0) returns NULL. Dividing total_revenue by NULL safely yields NULL rather than crashing your analytics pipeline.

3. Engine-Specific Fallback Functions

While I always recommend using the standard COALESCE() function for portability, you will frequently encounter database-specific alternatives in legacy codebases:

How NULL Behaves in Aggregate Functions

Aggregate functions (COUNT, SUM, AVG, MIN, MAX) treat NULL values in very specific ways that directly impact report calculations.

Table: sales_incentives
┌───────────────┬─────────────────┐
│ employee_id   │ incentive_bonus │
├───────────────┼─────────────────┤
│ 101           │ 1000.00         │
│ 102           │ 2000.00         │
│ 103           │ NULL            │
│ 104           │ 3000.00         │
└───────────────┴─────────────────┘

The Difference Between COUNT(*) and COUNT(column)

• COUNT(*): Counts every physical row returned by the query, including rows containing NULL values.
• COUNT(column_name): Counts only the rows where the specified column is NOT NULL.

SQL

-- Using the sample table above:
SELECT 
    COUNT(*) AS total_rows,                  -- Returns: 4
    COUNT(incentive_bonus) AS bonus_count    -- Returns: 3 (ignores row 103)
FROM sales_incentives;

Distortions in AVG() Calculations

All mathematical aggregates (SUM, AVG, MIN, MAX) automatically ignore NULL values. In statistical calculations, this can produce unexpected mathematical results:

SQL

-- Computes (1000 + 2000 + 3000) / 3 = 2000.00
SELECT AVG(incentive_bonus) AS avg_bonus_ignoring_null
FROM sales_incentives;

-- Computes (1000 + 2000 + 0 + 3000) / 4 = 1500.00
SELECT AVG(COALESCE(incentive_bonus, 0)) AS avg_bonus_including_all_staff
FROM sales_incentives;

If your business rule requires missing bonuses to count as zero, relying on default AVG() behavior will artificially inflate your metric by dividing by 3 instead of 4.

NULL Behavior in GROUP BY, ORDER BY, and DISTINCT

While NULL = NULL is UNKNOWN in boolean filters, SQL engines treat NULL values as equivalent grouping keys and distinct entities for set operations.

1. GROUP BY

When grouping by a column containing multiple NULL records, SQL consolidates all NULL values into a single aggregate group:

SQL

SELECT department_id, COUNT(*) AS headcount
FROM employees
GROUP BY department_id;

All employees with no assigned department are grouped together under one NULL header.

2. DISTINCT

The DISTINCT keyword treats multiple NULL rows as duplicates and returns a single NULL entry:

SQL

SELECT DISTINCT region_code
FROM warehouse_locations;

3. ORDER BY Sorting Rules

Because NULL is not a numerical quantity, database engines have differing conventions on whether NULL sorts as the highest or lowest value.

To guarantee consistent cross-platform sorting behavior regardless of the database engine, use the ANSI standard NULLS FIRST or NULLS LAST syntax:

SQL

-- Explicitly force missing values to the bottom regardless of sort direction
SELECT account_id, balance
FROM corporate_accounts
ORDER BY balance DESC NULLS LAST;

The Dangerous Trap: NOT IN with Subqueries Containing NULL

One of the most destructive pitfalls in SQL query design occurs when combining the NOT IN predicate with a dataset or subquery containing a NULL value.

The Problem Explained

Consider these two simple tables:

SQL

-- parent_departments (department_id: 10, 20, 30, 40)
-- inactive_departments (department_id: 30, NULL)

SELECT department_name 
FROM parent_departments
WHERE department_id NOT IN (SELECT department_id FROM inactive_departments);

You might expect this query to return departments 10, 20, and 40. Instead, it returns zero rows.

Why This Happens

SQL expands NOT IN (30, NULL) into sequential comparisons joined by AND:

SQL

WHERE (department_id != 30) AND (department_id != NULL)

For every row evaluated:

1. department_id != 30 evaluates to TRUE or FALSE.
2. department_id != NULL always evaluates to UNKNOWN.
3. TRUE AND UNKNOWN evaluates to UNKNOWN.

Because the predicate never evaluates to TRUE, the entire query returns an empty result set.

The Solution: Use NOT EXISTS

Always write negative membership checks using NOT EXISTS, which relies on boolean existence rather than set equality:

SQL

-- SAFE & ROBUST: Unaffected by NULL values in the target table
SELECT p.department_name
FROM parent_departments p
WHERE NOT EXISTS (
    SELECT 1 
    FROM inactive_departments i 
    WHERE i.department_id = p.department_id
);

Database Design: NOT NULL Constraints vs. Default Values

Handling NULL in queries adds overhead and complexity. Proper schema architecture minimizes unnecessary NULL columns from the start.

SQL

CREATE TABLE financial_accounts (
    account_id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    account_number VARCHAR(32) NOT NULL,
    account_balance NUMERIC(15, 2) NOT NULL DEFAULT 0.00,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
    closure_date TIMESTAMPTZ NULL -- Legitimate use: Account may still be open
);

When to Use NOT NULL with Defaults

• Status Flags: Use BOOLEAN NOT NULL DEFAULT FALSE rather than a nullable column with 3 states (TRUE, FALSE, NULL).
• Numeric Quantities: Use numeric columns defaulted to 0 or 0.00 if a missing balance truly represents zero money.
• Date Created / Timestamps: Always apply NOT NULL DEFAULT CURRENT_TIMESTAMP.

When NULL Is Appropriate

• Future or Unknown Events: An order_shipped_date or termination_date must remain NULL until the event actually occurs. Using fake “sentinel values” (like '1900-01-01' or '9999-12-31') creates technical debt and corrupts data validation.
• Optional Attributes: An optional suite_number in a mailing address table.

Best Practices Checklist for SQL NULL Handling

Keep this architectural checklist in mind when writing queries and designing relational schemas:

• [ ] Never Use = NULL or != NULL: Always use IS NULL or IS NOT NULL for filtering missing values.
• [ ] Guard Subqueries with NOT EXISTS: Avoid NOT IN against columns or subqueries that might contain NULL values.
• [ ] Standardize on COALESCE(): Use COALESCE() for fallback handling to ensure queries remain portable across database engines.
• [ ] Be Explicit in Aggregations: Determine whether AVG() should calculate over all rows (using COALESCE(col, 0)) or only existing values.
• [ ] Specify Sorting Order Explicitly: Use NULLS FIRST or NULLS LAST in ORDER BY clauses to eliminate engine-specific sort variations.
• [ ] Prevent Division by Zero with NULLIF(): Use NULLIF(denominator, 0) to gracefully return NULL instead of generating fatal runtime division errors.
• [ ] Enforce NOT NULL at the Schema Layer: If a column should never be missing, enforce a NOT NULL constraint at table creation rather than relying on application-layer validation.

You may also like the following articles:

• SQL Indexes
• SQL Server Views
• COALESCE SQL
• SQL Scalar Functions

SQL Indexes Best Practices

Understanding SQL Index Architecture: How Indexes Actually Work

Before diving into optimization strategies, we must understand the mechanics under the hood. At its core, an index is a persistent data structure (most commonly a balanced B-Tree) designed to minimize disk I/O by allowing the database engine to locate specific rows without scanning an entire table.

B-Tree Structure Breakdown

A standard B-Tree index consists of three distinct layers:

1. Root Node: The single entry point examined by the query engine to direct the search down the hierarchy.
2. Intermediate Nodes: Branching levels that hold key values and pointers to direct traversal to the next appropriate node level.
3. Leaf Nodes: The bottom layer of the tree. In a non-clustered index, leaf nodes contain the indexed column values paired with a row locator (a pointer or clustering key). In a clustered index, the leaf nodes are the actual data pages.

When a query requests data without a suitable index, the engine must perform a Full Table Scan (or Clustered Index Scan), reading every single page allocated to that table into memory. With a well-placed index, the engine executes an Index Seek, traversing the B-Tree directly to the target record in a fraction of the computational cost and time.

Clustered vs. Non-Clustered Indexes: The Strategic Difference

Choosing between clustered and non-clustered indexes is one of the first architectural decisions you make when defining a schema.

1. Clustered Indexes

A clustered index determines the physical order of data storage on disk. Because the data itself can only be sorted in one way, you can have only one clustered index per table.

• Primary Purpose: Organizing the base table pages.
• Storage Overhead: Zero additional storage beyond the table data pages and minimal B-Tree navigational structure.
• Optimal Selection: Surrogate identity keys, sequential primary keys (such as auto-increment integers), or strictly monotonic chronological timestamps.

2. Non-Clustered Indexes

A non-clustered index is a completely separate physical structure that contains the indexed key columns along with a bookmark (pointer) back to the base data row.

• Primary Purpose: Optimizing secondary search paths, filtering conditions, and join predicates.
• Storage Overhead: Substantial, as it duplicates the indexed column values in a distinct physical allocation.
• Optimal Selection: Foreign keys, frequently filtered status columns, lookup attributes, and columns participating in JOIN, ORDER BY, or GROUP BY clauses.

Best Practices for Choosing Index Key Columns

Selecting which columns to index requires a deliberate balance between query retrieval speed and data manipulation throughput.

1. Prioritize High Selectivity Columns

Selectivity measures how distinct the values are within a given column relative to the total row count.

$$\text{Selectivity} = \frac{\text{Number of Distinct Values}}{\text{Total Number of Rows}}$$

• High Selectivity (Close to 1.0): Ideal for indexing. Columns with unique identifiers, email addresses, or transaction reference codes allow the engine to eliminate massive percentages of data pages immediately.
• Low Selectivity (Close to 0.0): Poor candidates for standard B-Tree indexing. Columns with few distinct values (such as binary flags or status codes) typically prompt the query optimizer to abandon the index in favor of a scan unless filtered down using specialized techniques.

2. Match the Left-to-Right Rule in Composite Indexes

When creating multi-column (composite) indexes, column order dictates usability. The database engine traverses composite indexes following the “Left-to-Right Rule” (also known as the leftmost prefix rule).

Consider an index defined on (Column_A, Column_B, Column_C):

• Queries filtering by Column_A will utilize the index.
• Queries filtering by Column_A AND Column_B will utilize the index efficiently.
• Queries filtering by Column_A AND Column_B AND Column_C will achieve optimal seek performance.
• Queries filtering only by Column_B or Column_C cannot seek directly into the index B-Tree, forcing an index scan.

Place the most frequently filtered, equality-based (=) columns at the beginning of the composite key, followed by range-based (<, >, BETWEEN, LIKE) columns.

3. Leverage Covering Indexes with Included Columns

A Covering Index occurs when an index contains all the columns requested by a specific query (both in the SELECT list and the WHERE/JOIN clauses). When an index is fully covering, the database engine retrieves all necessary data directly from the index leaf pages, completely bypassing expensive Key Lookups or RID Lookups against the base table.

To build covering indexes without bloating the root and intermediate B-Tree nodes, use the INCLUDE clause:

• Key Columns: Participate in sorting, filtering, joining, and B-Tree navigation.
• Included Columns (INCLUDE): Stored only at the leaf level. They do not increase the depth or maintenance cost of intermediate B-Tree navigation but satisfy the SELECT projection.

Specialized Indexing Types and When to Deploy Them

Modern relational database engines offer specialized index variants tailored for distinct architectural patterns.

1. Filtered (Partial) Indexes

Filtered indexes include a WHERE predicate directly in the index definition, storing only a subset of table rows.

• Use Case: Tables with highly skewed distributions—such as indexing only non-processed records where status equals “Pending”, or indexing nullable columns where values are not null.
• Advantages: Radically smaller index footprints, faster maintenance overhead, and highly accurate statistics for the specific data slice.

2. Unique Indexes

Unique indexes enforce entity integrity at the storage engine level, guaranteeing that no two rows contain identical key values.

• Use Case: Natural keys, social security numbers, corporate tax identifiers, and system usernames.
• Performance Benefit: Aside from ensuring data integrity, unique indexes provide the optimizer with absolute cardinality guarantees, enabling deterministic execution plans and early-exit lookups.

3. Columnstore Indexes

Unlike traditional row-oriented B-Trees that store entire records together on data pages, Columnstore indexes organize and store data column by column.

• Use Case: Analytical workloads (OLAP), data warehousing, and aggregation queries running across millions or billions of rows (SUM, AVG, COUNT).
• Performance Benefit: High data compression ratios (often 5x to 10x) and massive reductions in disk I/O, as the engine reads only the specific columns requested in the query.

Anti-Patterns: Critical Indexing Mistakes to Avoid

In database tuning, knowing what not to do is just as important as knowing what to build. Below are the most damaging indexing mistakes in production systems:

1. Applying Functions on Indexed Columns (Non-SARGable Queries)

A query is SARGable (Search Argument Able) when the database engine can directly exploit an index seek. Wrapping an indexed column inside a scalar function, mathematical operation, or explicit type conversion forces the engine to evaluate that function for every single row, breaking the B-Tree seek mechanism.

• Problematic Construct: Wrapping date columns inside formatting functions or substring extractions on strings.
• Remediation: Rewrite the predicate so the column remains bare on one side of the operator, transforming the search argument into a clean range.

2. Leading Wildcards in String Searches

Using wildcard patterns at the beginning of a string search (such as pattern matching against %Term) prevents the database engine from navigating the B-Tree hierarchy from left to right. This immediately forces a full index scan.

• Remediation: Use trailing wildcards (Term%) where possible, or deploy Full-Text Search engines when arbitrary substring matching is mandatory.

3. Over-Indexing and the Write Penalty

Every non-clustered index created on a table is not free; it represents a live copy of that data. When an application executes an INSERT, UPDATE, or DELETE:

• The base table is modified.
• Every single non-clustered index containing the affected columns must be synchronously updated within the same transaction.

Over-indexing severely throttles write throughput, inflates transaction log generation, and increases concurrency locking and blocking issues.

Maintaining and Monitoring Index Health

Indexes are not “set-and-forget” objects. Over time, continuous data modifications degrade index quality and query performance.

1. Managing Fragmentation

Index fragmentation occurs when data modifications cause page splits, leaving pages half-empty or physically scattered out of logical order across storage disks.

• Internal Fragmentation: Unused space inside index pages resulting in wasted memory buffer pools and extra disk I/O.
• External Fragmentation: Physical allocation of pages does not match the logical B-Tree order.

Maintenance Strategy

• Low Fragmentation (< 10-15%): No action required.
• Moderate Fragmentation (15% to 30%): Perform an online index Reorganize (defragments leaf pages with minimal locking).
• High Fragmentation (> 30%): Perform an index Rebuild (drops and recreates the entire B-Tree structure, updating statistics).

2. Statistics Maintenance

The query optimizer relies on distribution statistics associated with indexes to estimate row counts and cost-effective execution paths. If statistics become stale due to high data turnover, the optimizer may choose a full table scan over an index seek, even when a perfect index exists. Ensure automated statistics updates are enabled and augment them with scheduled maintenance on volatile tables.

3. Auditing Unused and Duplicate Indexes

Regularly query dynamic management views and system catalogs to identify unused or redundant indexes.

• Duplicate Indexes: Indexes with identical key columns in the same order.
• Redundant Indexes: An index on (Column_A) when another composite index already exists on (Column_A, Column_B).
• Unused Indexes: Indexes with zero user seeks/scans over extended reporting periods but millions of maintenance writes.

Summary of Core Principles

Designing high-performance database systems requires an intentional, evidence-based approach to indexing:

1. Design for read-write balance: Every index accelerates reads while adding direct latency to writes. Index only what your query workload actively demands.
2. Respect the B-Tree hierarchy: Order composite keys methodically and keep search predicates SARGable.
3. Audit proactively: Continually monitor execution plans, eliminate unused indexes, and keep statistics fresh to ensure the optimizer consistently makes the right choices.

You may also like the following articles:

• SQL Indexes
• SQL Server Views
• How to Execute Function in SQL Server with Parameters

SQL Indexes

What Is a SQL Index?

At its simplest, a SQL index is an auxiliary on-disk data structure that the database engine uses to locate and retrieve rows significantly faster than scanning the entire table.

Think of an index like the index at the back of a comprehensive technical reference book. If you want to find every reference to “Connection Pooling,” you do not read the entire 900-page book front to back. You turn to the back, look up the term alphabetically, find the exact page numbers, and jump straight to those pages.

In SQL engines like Microsoft SQL Server, PostgreSQL, and MySQL, an index performs this exact operation on database storage pages.

SQL

-- Basic syntax for creating an index
CREATE INDEX idx_customers_lastname 
ON dbo.Customers (LastName);

After executing the above query, I got the expected output and the index got created successfully as shown in the screenshot below.

Check out SQL Indexes Best Practices

The Cost of Missing Indexes: Full Table Scans vs. Index Seeks

When a table lacks an index on a filtered column, the database engine must execute a Full Table Scan (or Table Scan / Clustered Index Scan). It loads every single page of data into memory buffer pools and evaluates the predicate row by row.

When an index exists, the engine performs an Index Seek. It traverses a tree structure directly to the qualifying rows with minimal disk input/output (I/O) operations.

How SQL Indexes Work Under the Hood: B-Trees Explained

Most relational database indexes are implemented using a B-Tree (Balanced Tree) data structure. Understanding the physical layout of a B-Tree is crucial for writing queries that effectively leverage the index.

A B-Tree index maintains a hierarchical tree structure with three distinct tiers:

1. Root Node: The single entry point at the top of the tree. It contains pointers and key ranges that guide searches to the next level down.
2. Intermediate Nodes: Branch levels between the root and leaves. These nodes contain sorted key ranges that direct the database engine to the exact child page holding the requested data.
3. Leaf Nodes: The bottom layer of the tree. In a nonclustered index, the leaf nodes contain the indexed key values along with a row locator (a pointer) to the actual physical row. In a clustered index, the leaf nodes are the actual data pages.

Because B-Trees are self-balancing, every leaf node sits at the exact same depth from the root. Whether querying a record that starts with “Adams” or “Zimmerman,” the database engine traverses the exact same number of pages (typically 3 to 5 levels deep, even for tables containing tens of millions of rows).

Primary Types of SQL Indexes

Relational database systems primarily organize data using two structural models: Clustered Indexes and Nonclustered Indexes.

1. Clustered Indexes

A clustered index defines the physical storage order of the data within a table. Because physical rows can only be sorted on disk in one order, a table can have only one clustered index.

• When you create a clustered index on a column, the leaf level of the B-Tree contains the actual data rows of the table.
• In engines like Microsoft SQL Server, creating a PRIMARY KEY constraint automatically builds a unique clustered index by default unless configured otherwise.
• A table without a clustered index is stored as an unordered structure known as a Heap.

SQL

-- Creating an explicit Clustered Index
CREATE CLUSTERED INDEX cdx_employees_employeeid 
ON dbo.Employees (EmployeeID);

After executing the above query, I got the expected output and the Clustered index got created successfully as shown in the screenshot below.

2. Nonclustered Indexes

A nonclustered index is a structure completely separate from the actual data rows. It contains the indexed column values sorted in order, paired with a row locator that tells the database engine where the base row lives.

• If the base table is a clustered table, the row locator is the Clustered Index Key.
• If the base table is a heap, the row locator is a physical Row Identifier (RID) pointing to file, page, and slot numbers.
• You can define multiple nonclustered indexes on a single table (typically up to 999 depending on the database engine, though production systems rarely need more than 5 to 10 per table).

SQL

-- Creating a standard Nonclustered Index
CREATE NONCLUSTERED INDEX idx_orders_orderdate 
ON dbo.Orders (OrderDate);

Clustered vs. Nonclustered Indexes: Structural Comparison

Specialized Index Types and Modern Patterns

Beyond standard single-column indexes, modern database engines offer specialized index patterns designed to solve specific query bottlenecks.

1. Composite Indexes (Multi-Column Indexes)

A composite index is an index built on two or more columns. The database engine sorts the data first by the primary column, then by the secondary column, and so on.

SQL

CREATE NONCLUSTERED INDEX idx_customers_state_city 
ON dbo.Customers (State, City);

The “Leftmost Prefix” Rule

When designing composite indexes, column ordering is critical. The database engine can only use an index if the query filters include the leftmost leading column in the index definition.

Given the index on (State, City):

• WHERE State = 'Texas' AND City = 'Austin' $\rightarrow$ Uses the Index (Full Seek)
• WHERE State = 'Texas' $\rightarrow$ Uses the Index (Prefix Seek)
• WHERE City = 'Austin' $\rightarrow$ Cannot Use the Index (Must scan because data is not sorted by City first)

2. Covering Indexes and the INCLUDE Clause

When a query requests columns that are not part of a nonclustered index, the engine must perform an index seek and then make an expensive jump to the base table to fetch the remaining columns. This secondary lookup is known as a Key Lookup or Bookmark Lookup.

To eliminate Key Lookups, you can design a Covering Index using the INCLUDE clause. Included columns are appended directly to the leaf nodes of the B-Tree without contributing to the sorting hierarchy of the intermediate nodes.

SQL

CREATE NONCLUSTERED INDEX idx_employees_department 
ON dbo.Employees (DepartmentID)
INCLUDE (FirstName, LastName, Salary);

With this index in place, a query selecting FirstName, LastName, and Salary filtered by DepartmentID resolves entirely within the index leaf pages, generating zero base table lookups.

3. Filtered / Partial Indexes

A filtered index (or partial index in PostgreSQL) indexes only a subset of rows meeting a defined predicate. This drastically reduces index size, maintenance overhead, and page count.

SQL

-- Indexing only active records to save space and speed up queries
CREATE NONCLUSTERED INDEX idx_orders_unprocessed 
ON dbo.Orders (OrderStatus, OrderDate)
WHERE OrderStatus IN ('Pending', 'Processing');

Filtered indexes are well suited for:

• Columns with heavily skewed data distributions (e.g., millions of Archived records vs. thousands of Active records).
• Sparse columns where the vast majority of values are NULL.

4. Unique Indexes

A unique index guarantees that no two rows in the indexed column contain identical values. While it behaves like a standard index for search optimizations, it doubles as an integrity constraint.

SQL

CREATE UNIQUE NONCLUSTERED INDEX udx_users_email 
ON dbo.Users (EmailAddress);

Best Practices for Designing High-Performance SQL Indexes

Over-indexing can degrade write performance just as severely as under-indexing degrades read performance. Follow these core design principles:

Choose Narrow, Static, and Sequential Clustered Keys

Because every nonclustered index stores a copy of the clustered index key as its row locator, wide clustered keys inflate the storage footprint of every other index on the table.

• Narrow: Use standard integer or BIGINT types over wide character strings.
• Sequential: Monotonically increasing values (such as IDENTITY or BIGINT GENERATED ALWAYS AS IDENTITY) append new records cleanly to the end of data pages, preventing costly page splits.
• Static: Avoid clustered keys on values that update frequently.

Index Foreign Keys Systematically

Relational engines do not automatically index foreign key columns when you declare a foreign key constraint. Manually indexing foreign keys improves the performance of JOIN operations and reduces blocking during cascading operations or parent-record deletions.

Place High-Selectivity Columns First in Composite Indexes

Selectivity measures how unique values are across a column. A column with high selectivity (such as SSN or TransactionID) narrows down the result set much faster than a column with low selectivity (such as Gender or StatusFlag).

$$\text{Selectivity} = \frac{\text{Number of Distinct Values}}{\text{Total Number of Rows}}$$

In composite indexes, place the highest selectivity columns at the leading position unless your primary query access patterns dictate otherwise.

Strategic Summary

Designing an optimal indexing layer requires balancing fast read paths with acceptable write latency. Every new index introduces write amplification, as every INSERT, UPDATE, and DELETE must modify the base table along with all associated index trees.

You may also like the following articles:

• SQL Normalization
• SQL NULL
• SQL Server Views
• SQL MIN MAX
• SQL COUNT DISTINCT

SQL Server Views

The Internal Mechanics of SQL Server Views

To utilize views effectively in production, you must first understand how the Microsoft SQL Server relational engine processes them behind the scenes.

When an application queries a standard view, SQL Server executes a multi-step resolution process:

1. Syntax Parsing and Object Binding: The parser identifies the view name in the sys.views and sys.objects catalog metadata and verifies that the calling user has appropriate permissions on the view.
2. View Expansion (Query Inlining): The query processor retrieves the compiled T-SQL definition from sys.sql_modules. It replaces the view reference in the outer query with the underlying query expression, integrating filter predicates and projection columns.
3. Query Optimization: The SQL Server Cost-Based Optimizer analyzes the expanded query tree alongside the underlying base tables, indexes, and statistics to generate an optimal physical execution plan.
4. Execution: The storage engine reads the required data pages directly from the physical base tables in the buffer cache or storage subsystem.

Because the relational engine dynamically folds the view definition into the user query, a standard view generally carries zero memory overhead when idle and introduces minimal compilation cost.

Why Use Views in Enterprise Database Architectures?

• Logical Data Independence: Views establish an abstraction layer between physical database structures and application layers. If a database administrator refactors a physical table by splitting it or renaming columns, the view can be modified to maintain the original schema signature, preventing downstream application failures.
• Granular Security and Row/Column Masking: Views enable secure data presentation without granting direct SELECT permissions on underlying base tables. Sensitive columns (such as tax identifiers or billing rates) can be excluded from the view projection, while row-level filtering can restrict user access based on organizational boundaries.
• Simplification of Complex Relational Logic: Enterprise queries frequently require multi-table INNER and OUTER JOIN operations, derived tables, and aggregation expressions. Encapsulating this logic inside a view allows software engineers and reporting tools to query the dataset as if it were a single, normalized table.
• Centralization of Business Rules: Embedding standardized calculations, state codes, and operational flags into view definitions ensures uniform reporting across multiple analytics platforms, reporting services, and custom applications.

Core Types of SQL Server Views

SQL Server provides distinct types of views tailored for specific architectural requirements. Choosing the correct view type is critical for balancing system performance, storage utilization, and query complexity.

1. Standard Views

Standard views are dynamic, non-materialized virtual tables. They store only metadata—the SELECT query definition—in the system catalog. The query runs every time the view is called, ensuring that users always access real-time data from the underlying tables.

2. Indexed Views (Materialized Views)

An indexed view is a view that has been physically materialized on disk by creating a unique clustered index on its result set. When the base tables are modified via INSERT, UPDATE, or DELETE operations, the SQL Server Database Engine automatically updates the data stored in the indexed view. These views accelerate aggregate-heavy and multi-join reporting queries.

3. Partitioned Views

Partitioned views stitch together horizontally split data from multiple tables across one or more databases, presenting the combined data as a single continuous result set using the UNION ALL operator.

• Local Partitioned Views: Combine tables residing on the same SQL Server instance.
• Distributed Partitioned Views: Combine tables across independent SQL Server instances linked via Linked Servers or distributed transactions, allowing horizontal scalability for massive datasets.

4. System Views and Dynamic Management Views (DMVs)

SQL Server exposes internal database engine metadata and runtime diagnostic metrics through system views, such as sys.tables, sys.indexes, and Dynamic Management Views (sys.dm_*). These catalog structures are maintained by the database engine to provide administrators with visibility into system health, query performance, and resource locking.

Comparative Architecture Matrix

The following table contrasts the primary implementation characteristics of the three main user-defined view architectures in SQL Server:

Creating and Managing Views: Syntax and Schema Options

Creating production-ready SQL Server views requires more than a simple CREATE VIEW statement. T-SQL provides several view definition clauses that control security, integrity, and dependency tracking.

Core Syntax and Declarative Clauses

A standard view is instantiated using the CREATE VIEW statement, with modification and removal handled by ALTER VIEW and DROP VIEW:

SQL

CREATE VIEW Sales.ActiveCustomerInvoices
WITH SCHEMABINDING, ENCRYPTION, VIEW_METADATA
AS
SELECT 
    c.CustomerID,
    c.CustomerName,
    i.InvoiceID,
    i.InvoiceDate,
    i.InvoiceTotal
FROM Sales.Customers AS c
INNER JOIN Sales.Invoices AS i 
    ON c.CustomerID = i.CustomerID
WHERE i.IsSettled = 0
WITH CHECK OPTION;

After executing the above query, the view got created successfully as shown in the screenshot below.

Advanced View Clauses Explained

Understanding these optional clauses is essential when designing reliable database solutions:

1. WITH SCHEMABINDING

Schema binding binds the view directly to the underlying physical schema of the referenced tables. When WITH SCHEMABINDING is active:

• Base tables cannot be modified using ALTER TABLE or DROP TABLE in any way that would break the view definition.
• All referenced objects must use two-part naming conventions (SchemaName.ObjectName).
• Base tables and referenced user-defined functions must exist in the same database.
• Crucial Prerequisite: Schema binding is mandatory if you plan to create a clustered index on the view.

2. WITH CHECK OPTION

When a view is used to perform data modifications (INSERT or UPDATE), WITH CHECK OPTION forces all modifications to comply with the filtering criteria defined in the view’s WHERE clause.

For instance, if a view filters for StateCode = 'TX', any INSERT or UPDATE executed through the view that attempts to assign a different state code will be rejected by the database engine. This prevents data from disappearing from the view immediately after an update.

3. WITH ENCRYPTION

The WITH ENCRYPTION clause obfuscates the view definition stored in the sys.sql_modules system catalog. This prevents non-administrative users and external tools from viewing the underlying SQL logic, protecting sensitive business logic and proprietary queries.

4. WITH VIEW_METADATA

This clause specifies that SQL Server will return view-level metadata rather than base-table metadata to the client application when browsing APIs (such as DB-Library, ODBC, or OLE DB) request schema information. This ensures front-end applications treat the view as an independent, primary entity.

Indexed Views: Deep Dive and Engine Requirements

Indexed views are one of the most powerful optimization mechanisms in SQL Server. By creating a unique clustered index on a view, you convert it from a dynamic query into a physically stored, high-performance dataset.

Determinism and Session Set Options

To create an indexed view, all expressions within the SELECT statement must be fully deterministic—meaning they must always return the exact same output for a given set of input values. Functions like GETDATE(), NEWID(), or RAND() cannot be used in indexed views.

Furthermore, indexed views require specific session-level SET options during creation and subsequent DML operations:

• QUOTED_IDENTIFIER ON
• ANSI_NULLS ON
• ANSI_PADDING ON
• ANSI_WARNINGS ON
• ARITHABORT ON
• CONCAT_NULL_YIELDS_NULL ON
• NUMERIC_ROUNDABORT OFF

Structural Restrictions on Indexed Views

To guarantee that the physical clustered index can be maintained efficiently, SQL Server enforces strict structural rules:

• Must be defined WITH SCHEMABINDING.
• Must not contain OUTER JOIN, UNION, DISTINCT, TOP, or subqueries.
• If GROUP BY is utilized, the SELECT list must include COUNT_BIG(*).
• Aggregations cannot use AVG(); you must store SUM() and COUNT_BIG() separately and compute the average in your application query.

Updatable Views and DML Constraints

A common point of confusion among developers is whether data can be inserted, updated, or deleted through a view. SQL Server does allow direct Data Manipulation Language (DML) operations through views, provided the operations satisfy strict engine constraints:

• Single Base Table Constraint: Any INSERT, UPDATE, or DELETE statement executed against a view can modify data in only one underlying base table at a time. Multi-table modifications in a single statement are rejected.
• Non-Derivation Rule: Columns targeted for update must map directly to raw base table columns. You cannot update columns that rely on calculated expressions, string concatenations, or aggregate functions.
• Nullable and Default Column Integrity: Any INSERT statement executed through a view must supply values for all non-nullable columns in the base table that lack a default constraint, even if those columns are omitted from the view projection.

When an application must perform multi-table writes through a single view interface, the standard solution is to attach an INSTEAD OF Trigger to the view. The trigger intercepts incoming INSERT, UPDATE, or DELETE commands, allowing you to route the write operations across multiple underlying base tables using custom T-SQL transactions.

Performance Considerations and the View-on-View Anti-Pattern

While views provide clear architectural advantages, improper design can introduce severe performance bottlenecks in enterprise SQL Server databases.

1. The “View-on-View” Nesting Anti-Pattern

One of the most frequent performance issues I encounter in production systems is deep view nesting—the practice of building views that query other views, which in turn query additional views.

When SQL Server’s query optimizer attempts to expand nested views that are 4 to 6 layers deep, several issues arise:

• Query Graph Explosion: The expanded query tree becomes massive, consuming significant CPU compilation time.
• Inaccurate Cardinality Estimation: The optimizer struggles to calculate accurate cardinality estimates across layered abstractions, often leading to poor execution plan choices, such as selecting inappropriate physical join operators or generating insufficient memory grants.
• Hidden Join Overhead: Downstream queries often process unnecessary joins and columns embedded deep within the view hierarchy, reading large amounts of unnecessary data from disk.

2. The NOEXPAND Query Hint for Indexed Views

In the Enterprise Edition of SQL Server, the query optimizer automatically searches for matching indexed views and substitutes them into an execution plan—even if the incoming query targets the base tables directly.

In Standard Edition, however, the optimizer does not automatically substitute indexed views. To force SQL Server to use the materialized clustered index rather than re-evaluating the underlying query against the base tables, you must explicitly use the WITH (NOEXPAND) table hint:

SQL

SELECT 
    CustomerID,
    TotalRevenue
FROM Sales.IndexedCustomerSummary WITH (NOEXPAND)
WHERE StateCode = 'NY';

Even in Enterprise Edition, I recommend applying WITH (NOEXPAND) in latency-critical workloads, as it bypasses the optimizer’s view-matching phase and guarantees that SQL Server reads the pre-computed clustered index pages directly.

Enterprise View Security: Ownership Chaining

Views are a cornerstone of secure database multi-tenancy because of Ownership Chaining.

When a user queries a view that references a base table, and both the view and the table share the same owner (such as the default dbo schema owner), SQL Server evaluates permissions only on the view. It skips permission checks on the underlying base tables.

This security model allows you to revoke all direct SELECT, INSERT, UPDATE, and DELETE permissions on physical tables from application accounts, routing all data access through managed views that enforce column projection limits and row filters.

Best Practices Checklist for Database Administrators

To maintain a secure, high-performance SQL Server environment, use this operational checklist when building and managing views:

• Enforce Two-Part Naming: Always declare object references using two-part notation (SchemaName.ObjectName) to support schema binding and avoid ambiguous resolution overhead.
• Avoid SELECT * in View Definitions: Explicitly declare every column in the projection list. Using SELECT * can lead to metadata synchronization errors when the underlying table schema changes, and it prevents the use of WITH SCHEMABINDING.
• Apply WITH SCHEMABINDING to Core Architectural Views: Use schema binding on foundational views to prevent unexpected table alterations from breaking production services.
• Keep View Nesting Under Two Levels: Limit view hierarchies to a depth of two to ensure predictable query execution plans and accurate cardinality estimates.
• Monitor Indexed View Maintenance Overhead: Before creating an indexed view, evaluate the write-to-read ratio of the base tables. Highly volatile tables with frequent INSERT and UPDATE traffic may suffer write-throughput penalties if they support multiple indexed views.
• Use WITH CHECK OPTION on Updatable Views: Prevent data corruption by verifying that all modifications performed through views conform to the view’s filtering rules.
• Refresh View Metadata After Base Table Modifications: If a view is not schema-bound and an underlying table is updated, run sp_refreshview to synchronize the view’s internal metadata with the new physical schema.

Summary

SQL Server views are an essential tool for building robust, secure, and scalable database architectures. By abstracting raw relational tables into clean virtual datasets, views protect data integrity, simplify complex application queries, and establish clear security boundaries.

Understanding the differences between dynamic standard views, physically materialized indexed views, and partitioned views allows you to balance query performance with storage and write-maintenance costs across your database environment.

You may also like the following articles:

• SQL Indexes
• SQL Scalar Functions
• SQL Database Design Best Practices
• SQL Join Basics
• SQL Subquery

SQL MIN MAX

What Are the SQL MIN() and MAX() Functions?

The MIN() and MAX() functions are ANSI SQL-compliant aggregate functions designed to evaluate a set of values in a specific column or expression and return a single scalar output:

• MIN(expression): Returns the minimum (lowest) value in a set.
• MAX(expression): Returns the maximum (highest) value in a set.

Both functions are universal across all relational database management systems (RDBMS), including Microsoft SQL Server, PostgreSQL, MySQL, Oracle Database, Snowflake, and Google BigQuery.

Because they are aggregate functions, they summarize multi-row inputs into a single row unless paired with an analytical OVER() clause or grouped across dimensional attributes using GROUP BY.

Syntax and Core Mechanics

The syntax for both functions is straightforward:

SQL

SELECT 
    MIN(column_name) AS lowest_value,
    MAX(column_name) AS highest_value
FROM table_name
WHERE filter_conditions;

Basic Aggregate Query

Consider a table named Corporate.EmployeeCompensation containing employee payroll data:

SQL

SELECT 
    MIN(BaseSalary) AS MinimumSalary,
    MAX(BaseSalary) AS MaximumSalary,
    MAX(BaseSalary) - MIN(BaseSalary) AS SalarySpread
FROM Corporate.EmployeeCompensation
WHERE EmploymentStatus = 'Active';

Result: After executing the above query, I got the expected output as shown in the screenshot below.

In this single pass, the database engine scans the filtered rowset, discards non-matching rows, ignores any unpopulated records, and computes both boundary points alongside the calculated dynamic range (SalarySpread).

How MIN() and MAX() Operate on Different Data Types

A common misconception among early-career developers is that MIN() and MAX() are strictly arithmetic tools. In standard relational database architecture, these functions operate across all sortable data types, including numeric, date/time, and character string columns.

1. Numeric Data Types (INT, BIGINT, DECIMAL, FLOAT)

With numeric types, the functions perform standard mathematical comparisons, accounting for positive and negative values:

SQL

SELECT 
    MIN(AccountBalance) AS DeepestOverdraft,
    MAX(AccountBalance) AS PeakLiquidity
FROM Finance.CommercialAccounts;

2. Date and Time Data Types (DATE, DATETIME2, TIMESTAMP)

When applied to temporal fields, MIN() and MAX() identify chronologically extreme timestamps:

• MIN(date_column): Identifies the earliest or oldest timestamp (furthest in the past).
• MAX(date_column): Identifies the latest or most recent timestamp (closest to current time or furthest into the future).

SQL

SELECT 
    MIN(CreationTimestamp) AS FirstSystemUserCreated,
    MAX(LastLoginTimestamp) AS MostRecentActivity
FROM Security.UserAuditLog;

3. Character and String Data Types (VARCHAR, CHAR, TEXT)

When applied to text, MIN() and MAX() evaluate values based on the database’s configured collation and character set encoding (e.g., ASCII, UTF-8, or Latin1):

• MIN(string_column): Returns the string that appears first in alphabetical/lexicographical order.
• MAX(string_column): Returns the string that appears last in alphabetical/lexicographical order.

SQL

SELECT 
    MIN(LastName) AS AlphabeticallyFirst,
    MAX(LastName) AS AlphabeticallyLast
FROM HumanResources.StaffDirectory;

Data Type Behavior Reference Table

Handling NULL Values and Three-Valued Logic

In relational databases governed by three-valued logic (True, False, Unknown), NULL represents the absence of a value.

When evaluating data sets, MIN() and MAX() strictly adhere to the ANSI SQL standard rule: aggregate functions automatically eliminate NULL values from the calculation.

SQL

-- Sample values in BonusTier column: [ 1000.00, NULL, 5000.00, NULL, 2500.00 ]
SELECT 
    MIN(BonusTier) AS MinBonus,
    MAX(BonusTier) AS MaxBonus
FROM Sales.CompensationPlan;

• MIN(BonusTier) evaluates only [1000.00, 5000.00, 2500.00], yielding 1000.00.
• MAX(BonusTier) evaluates the same subset, yielding 5000.00.
• The NULL values are excluded without raising a runtime warning or converting into zeros.

After executing the above query, I got the expected output as shown in the screenshot below.

The All-NULL Edge Case

If a column contains only NULL records, or if the WHERE clause filters out every row in the table, both MIN() and MAX() return NULL.

SQL

-- When no rows match criteria:
SELECT 
    MIN(AnnualBonus) AS LowestBonus,
    MAX(AnnualBonus) AS HighestBonus
FROM Sales.CompensationPlan
WHERE DepartmentID = 99999; -- Non-existent department

Result:

Plaintext

LowestBonus | HighestBonus
------------+-------------
NULL        | NULL

Production Tip: If your downstream application tier or API contract cannot accept a NULL response, wrap your aggregate function in a COALESCE() or ISNULL() expression to provide a deterministic fallback:

SQL

SELECT COALESCE(MAX(AnnualBonus), 0.00) AS SafeMaxBonus 
FROM Sales.CompensationPlan;

Categorical Aggregation: Combining MIN and MAX with GROUP BY

In enterprise reporting, you rarely need the global extreme across an entire table. Instead, you need extreme values grouped across business dimensions—such as regions, departments, or product categories.

When combined with the GROUP BY clause, MIN() and MAX() calculate the extreme boundaries independently for each distinct group partition.

Grouped Aggregation Example

SQL

SELECT 
    DepartmentName,
    StateLocation,
    COUNT(EmployeeID) AS TotalHeadcount,
    MIN(BaseSalary) AS DepartmentFloorSalary,
    MAX(BaseSalary) AS DepartmentCeilingSalary,
    MAX(HireDate) AS NewestTeamMemberHireDate
FROM Enterprise.PersonnelDirectory
GROUP BY 
    DepartmentName, 
    StateLocation
ORDER BY 
    DepartmentName ASC, 
    StateLocation ASC;

In this query, SQL Server partitions the records by unique combinations of DepartmentName and StateLocation, computing the discrete salary bounds and the latest hire date for each individual subset.

Filtering Aggregated Results Using the HAVING Clause

A frequent mistake in SQL development is confusing row-level filters (WHERE) with aggregate-level filters (HAVING).

• WHERE: Filters raw records before aggregations are calculated.
• HAVING: Filters aggregated result groups after the MIN() or MAX() evaluations have been computed.

SQL

-- INCORRECT: Aggregates are not allowed in the WHERE clause
-- SELECT DepartmentID, MAX(BaseSalary) FROM Payroll WHERE MAX(BaseSalary) > 100000 GROUP BY DepartmentID;

-- CORRECT: Using HAVING to filter aggregate boundaries
SELECT 
    DepartmentID,
    MIN(BaseSalary) AS LowestPay,
    MAX(BaseSalary) AS HighestPay
FROM Enterprise.Payroll
GROUP BY DepartmentID
HAVING MAX(BaseSalary) >= 125000.00
   AND MIN(BaseSalary) >= 50000.00;

The database engine first reads the table, calculates the minimum and maximum salaries per DepartmentID, and then discards any department where the highest salary is under $125,000 or the lowest salary is below $50,000.

Advanced Analytical Patterns: MIN() and MAX() as Window Functions

When invoked with an OVER() clause, MIN() and MAX() transform from standard aggregate functions into analytical window functions. Instead of collapsing multiple rows into one, they compute dynamic boundaries while preserving individual row identities.

1. Partition-Wide Boundaries Without Collapsing Rows

You can display an employee’s salary right next to their department’s maximum and minimum salary for immediate comparative analysis:

SQL

SELECT 
    EmployeeID,
    DepartmentID,
    LastName,
    BaseSalary,
    MIN(BaseSalary) OVER (PARTITION BY DepartmentID) AS DepartmentFloor,
    MAX(BaseSalary) OVER (PARTITION BY DepartmentID) AS DepartmentCeiling,
    BaseSalary - MIN(BaseSalary) OVER (PARTITION BY DepartmentID) AS DistanceFromFloor
FROM Enterprise.Payroll;

2. Cumulative / Running Extremes Over Time

By adding an ORDER BY clause inside the OVER() declaration, you can calculate a running minimum or running maximum across chronological records:

SQL

SELECT 
    TransactionDate,
    DailyRevenue,
    -- Tracks the highest single-day revenue achieved up to the current row
    MAX(DailyRevenue) OVER (
        ORDER BY TransactionDate 
        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS AllTimePeakRevenueToDate
FROM Sales.DailyFinancialLedger;

Frequently Asked Questions (FAQs)

Can I use DISTINCT inside MIN() or MAX()?

Yes, syntax like MIN(DISTINCT Column) is valid SQL. However, using DISTINCT inside MIN() or MAX() is computationally redundant. The minimum or maximum of a unique set of numbers is identical to the minimum or maximum of a duplicate set (e.g., MIN([5, 5, 10]) = 5 and MIN([5, 10]) = 5). Omitting DISTINCT avoids unnecessary sorting overhead.

Is SELECT MAX(ID) + 1 a safe way to generate primary keys?

No. This is a dangerous database anti-pattern. In multi-user concurrent systems, two transactions executing MAX(ID) + 1 simultaneously will read the same maximum number and attempt to insert identical keys, causing primary key collision errors or race conditions. Always use native database sequencing mechanisms like IDENTITY (SQL Server), AUTO_INCREMENT (MySQL), SERIAL/GENERATED ALWAYS AS IDENTITY (PostgreSQL), or SEQUENCE objects.

Can MIN() and MAX() evaluate multiple columns simultaneously?

Standard aggregate MIN() and MAX() operate vertically on a single column across multiple rows. If you need to evaluate the minimum or maximum value horizontally across multiple columns in a single row, use the ANSI standard LEAST() and GREATEST() scalar functions:

SQL

SELECT 
    ProductID,
    WarehouseA_Stock,
    WarehouseB_Stock,
    LEAST(WarehouseA_Stock, WarehouseB_Stock) AS LowestLocalInventory,
    GREATEST(WarehouseA_Stock, WarehouseB_Stock) AS HighestLocalInventory
FROM Inventory.StockLevels;

Summary and Key Takeaways

The SQL MIN() and MAX() functions are essential tools for extracting boundary insights from relational datasets:

• Universal ANSI Support: Compatible across all modern relational engines and cloud data warehouses.
• Broad Data Type Capability: Evaluates numbers mathematically, dates chronologically, and text strings lexicographically based on database collation.
• Automatic NULL Handling: Discards NULL values automatically without corrupting mathematical or statistical evaluations.
• Full-Row Retrieval: Use Window Functions (DENSE_RANK() OVER (...) = 1) inside a CTE to retrieve complete records matching extreme boundaries cleanly.

You may also like the following articles:

• How to Execute Function in SQL Server with Parameters
• COALESCE SQL
• SQL COUNT DISTINCT
• SQL Scalar Functions

How to Execute Function in SQL Server with Parameters

Types of Parameterized Functions in SQL Server

Before executing a function, you must identify its architectural type. SQL Server categorizes user-defined functions into three distinct varieties, each requiring a specific execution pattern:

• Scalar User-Defined Functions: Accept zero or more parameters and return a single, discrete data value (such as an INT, VARCHAR, DECIMAL, or DATETIME).
• Inline Table-Valued Functions (iTVFs): Accept parameters and return the result of a single SELECT statement formatted as a relational table structure. They contain no procedural BEGIN...END block.
• Multi-Statement Table-Valued Functions (MSTVFs): Accept parameters, populate an explicitly declared table variable using procedural code, and return that tabular result set.

The Two-Part Naming Rule: Why Schema Matters

The single most common error developers face when executing scalar functions in SQL Server is omitting the schema qualifier.

When calling built-in system functions (like GETDATE(), LEN(), or DATEADD()), SQL Server resolves the function name globally without a schema prefix. However, when executing a user-defined scalar function, SQL Server strictly requires a two-part name (schema_name.function_name).

SQL

-- Fails with: 'Cannot find either column "CalculateAnnualTax" or the user-defined function...'
SELECT CalculateAnnualTax(85000.00, 0.07);

-- Executes Successfully
SELECT dbo.CalculateAnnualTax(85000.00, 0.07);

Rule of Thumb: Always qualify your user-defined scalar functions with their associated schema (most commonly dbo., or custom schemas such as finance. or hr.). While Table-Valued Functions can sometimes resolve without the schema prefix, applying the two-part naming convention uniformly across all functions is an industry best practice.

Executing Scalar Functions with Parameters

Let’s assume we have created a scalar function named dbo.CalculateOvertimePay that accepts two parameters: @HourlyRate (DECIMAL(10,2)) and @HoursWorked (DECIMAL(5,2)), returning the total overtime payout as a DECIMAL(10,2).

SQL

CREATE FUNCTION dbo.CalculateOvertimePay
(
    @HourlyRate DECIMAL(10,2),
    @HoursWorked DECIMAL(5,2)
)
RETURNS DECIMAL(10,2)
AS
BEGIN
    DECLARE @OvertimeHours DECIMAL(5,2);
    DECLARE @OvertimePay DECIMAL(10,2) = 0.00;

    IF @HoursWorked > 40.00
    BEGIN
        SET @OvertimeHours = @HoursWorked - 40.00;
        SET @OvertimePay = @OvertimeHours * (@HourlyRate * 1.5);
    END

    RETURN @OvertimePay;
END;

After executing the above query, the function has been created successfully as shown in the screenshot below.

Here are the standard ways to execute this parameterized function:

Method 1: Direct SELECT Statement with Literal Parameters

The simplest way to call a scalar function is by passing hardcoded literal values inside a SELECT statement:

SQL

SELECT dbo.CalculateOvertimePay(35.50, 48.00) AS OvertimeCompensation;

Result: After executing the above query, I got the expected output as shown in the below screenshot.

Method 2: Passing Declared T-SQL Variables

In production scripts, stored procedures, or automation routines, you frequently pass local T-SQL variables into the function parameters:

SQL

DECLARE @EmployeeRate DECIMAL(10,2) = 42.00;
DECLARE @TotalHours DECIMAL(5,2) = 52.50;
DECLARE @CalculatedPayout DECIMAL(10,2);

-- Execute and assign to a local variable
SET @CalculatedPayout = dbo.CalculateOvertimePay(@EmployeeRate, @TotalHours);

-- Display the result
SELECT @CalculatedPayout AS TotalPayout;

After executing the above query, I got the expected output as shown in the below screenshot.

Method 3: Passing Column Values in Query Projections

You can execute a parameterized scalar function on a per-row basis by passing table column names as the arguments:

SQL

SELECT 
    EmployeeID,
    FirstName,
    LastName,
    BaseHourlyRate,
    WeeklyHoursWorked,
    dbo.CalculateOvertimePay(BaseHourlyRate, WeeklyHoursWorked) AS OvertimePay
FROM HumanResources.PayrollRoster
WHERE DepartmentID = 104;

During execution, SQL Server evaluates dbo.CalculateOvertimePay for each row returned by the FROM and WHERE clauses, injecting the values of BaseHourlyRate and WeeklyHoursWorked dynamically.

Method 4: Capturing Return Values Using the EXECUTE Keyword

While scalar functions are typically invoked inside expressions, you can also execute them using the EXEC or EXECUTE command combined with a return variable:

SQL

DECLARE @Result DECIMAL(10,2);

EXEC @Result = dbo.CalculateOvertimePay 
    @HourlyRate = 50.00, 
    @HoursWorked = 45.00;

SELECT @Result AS ExecutionResult;

After executing the above query, I got the expected output as shown in the below screenshot.

Executing Inline Table-Valued Functions (iTVFs) with Parameters

Inline Table-Valued Functions do not return a single scalar value; they return an entire relational rowset. Because they behave like parameterized views, you execute them inside the FROM clause of a SELECT statement just like a standard table.

Let’s look at an iTVF that retrieves active customer orders based on state and minimum order total:

SQL

CREATE FUNCTION Sales.GetCustomerOrdersByState
(
    @StateCode CHAR(2),
    @MinOrderAmount DECIMAL(10,2)
)
RETURNS TABLE
AS
RETURN
(
    SELECT 
        OrderID,
        CustomerID,
        OrderDate,
        TotalDue,
        ShipState
    FROM Sales.OrdersHeader
    WHERE ShipState = @StateCode
      AND TotalDue >= @MinOrderAmount
);

Executing the iTVF with Parameter Values

To execute this function, call it directly in the FROM clause, passing the required arguments inside parentheses:

SQL

SELECT 
    OrderID,
    CustomerID,
    OrderDate,
    TotalDue
FROM Sales.GetCustomerOrdersByState('TX', 500.00)
ORDER BY TotalDue DESC;

Joining an iTVF with Other Physical Tables

Because an iTVF returns a valid table structure, you can join its output to other database tables, apply aliases, and include additional filtering predicates:

SQL

SELECT 
    ord.OrderID,
    cust.CustomerName,
    cust.AccountTier,
    ord.TotalDue
FROM Sales.GetCustomerOrdersByState('CA', 1000.00) AS ord
INNER JOIN Sales.CustomerProfiles AS cust
    ON ord.CustomerID = cust.CustomerID
WHERE cust.AccountTier = 'Enterprise';

Executing Functions Dynamically Using CROSS APPLY and OUTER APPLY

What if you have a table of records and need to execute a Table-Valued Function for every row in that table, passing a column from the outer table into the function’s parameters?

Standard SQL INNER JOIN syntax cannot pass values dynamically into a table function parameter. To accomplish this, SQL Server provides the APPLY operator.

1. CROSS APPLY (Equivalent to an INNER JOIN)

CROSS APPLY invokes the table-valued function for each row of the outer table. If the function returns an empty result set for a given row, that outer row is excluded from the final output.

SQL

SELECT 
    cust.CustomerID,
    cust.CustomerName,
    cust.StateCode,
    ord.OrderID,
    ord.TotalDue
FROM Sales.CustomerProfiles AS cust
CROSS APPLY Sales.GetCustomerOrdersByState(cust.StateCode, 250.00) AS ord;

2. OUTER APPLY (Equivalent to a LEFT OUTER JOIN)

OUTER APPLY invokes the function for every outer row, but if the function returns no records, the outer row is still preserved with NULL values in the function’s columns:

SQL

SELECT 
    cust.CustomerID,
    cust.CustomerName,
    cust.StateCode,
    ord.OrderID,
    ord.TotalDue
FROM Sales.CustomerProfiles AS cust
OUTER APPLY Sales.GetCustomerOrdersByState(cust.StateCode, 250.00) AS ord;

Executing Multi-Statement Table-Valued Functions (MSTVFs)

Multi-Statement Table-Valued Functions define an explicit table variable layout within their signature and use procedural logic (IF...ELSE, loops, cursor operations) to populate that table before returning it.

SQL

CREATE FUNCTION HumanResources.GetDepartmentSalarySummary
(
    @DepartmentID INT
)
RETURNS @DepartmentSummary TABLE
(
    SummaryID INT IDENTITY(1,1) PRIMARY KEY,
    DepartmentID INT,
    Headcount INT,
    AverageSalary DECIMAL(12,2),
    TotalPayroll DECIMAL(14,2)
)
AS
BEGIN
    INSERT INTO @DepartmentSummary (DepartmentID, Headcount, AverageSalary, TotalPayroll)
    SELECT 
        DepartmentID,
        COUNT(EmployeeID),
        AVG(AnnualSalary),
        SUM(AnnualSalary)
    FROM HumanResources.EmployeeSalaries
    WHERE DepartmentID = @DepartmentID
    GROUP BY DepartmentID;

    RETURN;
END;

Executing the MSTVF

Just like an inline TVF, you execute an MSTVF within the FROM clause:

SQL

SELECT 
    DepartmentID,
    Headcount,
    AverageSalary,
    TotalPayroll
FROM HumanResources.GetDepartmentSalarySummary(102);

Working with Default and Optional Parameters

SQL Server functions support default parameters, but their execution mechanics differ significantly from stored procedures.

When executing a Stored Procedure, you can simply omit an argument if it has a defined default. However, when executing a User-Defined Function, you cannot omit the parameter. You must explicitly pass the DEFAULT keyword in the parameter’s position.

Function Definition with Defaults:

SQL

CREATE FUNCTION dbo.CalculateProductDiscount
(
    @ListPrice DECIMAL(10,2),
    @DiscountPercent DECIMAL(4,2) = 0.05 -- Default 5%
)
RETURNS DECIMAL(10,2)
AS
BEGIN
    RETURN @ListPrice - (@ListPrice * @DiscountPercent);
END;

Correct vs. Incorrect Execution Syntax:

SQL

-- INCORRECT: Throws an error (An invalid parameter was passed)
SELECT dbo.CalculateProductDiscount(100.00);

-- CORRECT: Using explicit DEFAULT keyword
SELECT dbo.CalculateProductDiscount(100.00, DEFAULT) AS DiscountedPrice;

-- CORRECT: Overriding the default value
SELECT dbo.CalculateProductDiscount(100.00, 0.15) AS DiscountedPrice;

Troubleshooting Common Errors

Error 1: “Cannot find either column or user-defined function…”

• Cause: The scalar function was called without the schema prefix (e.g., calling CalculateTax(100) instead of dbo.CalculateTax(100)).
• Fix: Prefix the function call with its schema name: dbo.CalculateTax(...).

Error 2: “The formal parameter ‘@ParamName’ was not supplied…”

• Cause: A parameter was completely omitted during the function invocation.
• Fix: Provide all parameters defined in the signature. If the parameter has a default value assigned, pass the literal keyword DEFAULT.

Error 3: “Invalid use of a side-effecting operator within a function…”

• Cause: Attempting to call non-deterministic system functions like NEWID() or RAND() or executing data modification statements (INSERT, UPDATE, DELETE to physical tables) inside the function.
• Fix: Remove side-effecting operations. For GUID generation or randomized data, pass those generated values into the function as parameters rather than generating them inside the function body.

Performance Best Practices for Parameterized Functions

1. Favor Inline TVFs Over Scalar Functions:Prior to SQL Server 2019, scalar UDFs forced queries into iterative, row-by-row execution (RBAR – Row By Agonizing Row), disabling parallel execution plans. Whenever possible, rewrite complex scalar functions as single-statement Inline Table-Valued Functions and join them with CROSS APPLY.
2. Leverage Scalar UDF Inlining (SQL Server 2019+):If you are running SQL Server 2019 (15.x) or higher with database compatibility level 150+, the query optimizer automatically inlines many scalar functions into the calling query execution plan, substantially reducing CPU overhead.
3. Ensure Parameter Data Types Match Exactly:Passing an NVARCHAR string into a function expecting a VARCHAR parameter forces implicit data type conversion during execution. This causes unnecessary CPU cycles and can prevent the query optimizer from leveraging existing column indexes on underlying tables.

Mastering the execution of parameterized functions in SQL Server is essential for writing clean, modular, and high-performing database routines.

You may also like the following articles:

• SQL Scalar Functions
• SQL Database Design Best Practices
• SQL Constraints
• SQL Join Basics

A COALESCE() function is the one way to deal with null values; here, you will understand what the COALESCE() function is with an example, and then you will understand the syntax of the COALESCE() function, which explains how to use it in your query.

COALESCE SQL

What is SQL Server COALESCE function?

SQL Server COALESCE function returns the first non-null value from the given list of values, which means if you have a list of values, and some of the values in the list are null, then this function ignores the null values and returns a first non-null value.

For example, if you have a list of numbers like this [null, null, 5, 3, 7], then the COALESCE() function returns the first non-null value, which is 5 in this case. It ignores the first two null values in the list.

After the first two null values in the list, the first value which is not null is 5. If you provide a list like this [null, 3, null, 5, 7], the first non-null value is 3, so the function will return this value.

The syntax is given below.

COALESCE(exp_1, exp_2, .., exp_3)

Where,

• COALESCE(): The function accepts a set of values containing null values and returns the first non-null value.
• exp_1, exp_2,.., exp_3: It is the expression evaluated by the COALESCE() function. It is ignored if the expression is null; otherwise, if it is the first non-null value, that expression is returned.

For example, you have a list of values like (null, ‘USA’, null, ‘New York’, ‘Canada’). To find the first non-null value from the list, you can use COALESCE(), as shown below.

SELECT 
	COALESCE(null, 'USA', null, 'New York', 'Canada') AS FristNonNullValue;

After executing the above query, I got the expected output as shown in the screenshot below.

The first non-null value is ‘USA’ in the list; here, you need to understand two things: first, look at the number of null values, and second, which non-null values are returned.

Here, in the query part, COALESCE(null, ‘USA’, null, ‘New York’, ‘Canada’), the COALESCE() function starts with the first value, which is the first null value, so it ignores this value. It moves to the next value, ‘USA’, which is the non-null value, the first non-null value in the list, so the function returns this value.

So, the COALESCE() ignores all the null values, and as soon as it finds any non-null values in the list, it returns that value and doesn’t care about other non-null values. For example, we have two null values in the list, ‘New York’ and ‘Canad’.

This is how the SQL Server COALESCE() function works.

Let’s move and see how to handle missing data in a table.

Handling Null Values in Table using SQL Server COALESCE Function

The table often contains null values. When any operation on tables yields results, some decisions are made using these results. What will happen if the result contains the wrong data? Will the decision also be wrong or ineffective? Yes, it affects the decision.

If the table contains null values while computing, it can ruin the results, which leads to bad decisions. So, there should be a way to handle these null values; here, you can use the COALESCE() function.

For example, a Customer table with columns CustomerID, Email and PhoneNumber is shown below.

As you can see, the table columns Email and PhoneNumber have some missing values for the customer.

Now, you are on a marketing campaign and have access to a database that stores information about customers with their email and phone numbers.

As you can see, some records are missing values in the Email and PhoneNumber columns. You have to write a query that ensures the campaign reaches them through an alternative contact method; here, you can use the COALESCE() function, as shown below.

SELECT
	CustomerID,
	COALESCE(Email, PhoneNumber) AS ContactInfo
FROM
	Customers;

After executing the above query, I got the expected output as shown in the screenshot below.

As you can see in the result set, the phone number is used wherever email is missing. For example, a customer with an ID equal to 2 emails is missing, so a phone number is used instead of email.

Let’s take another example where you must perform calculations or aggregate data where NULL values can affect the result or outcome.

For example, you have an OrderDetails table with columns, OrderID, ItemPrice, and Quantity, as shown below.

Here in the table, some of the item prices are missing in the ItemPrice column, and you need to compute the total value of the order when some items might not have a set price.

You can use the COALESCE() function, as shown below.

SELECT
	OrderID,
	SUM(COALESCE(ItemPrice, 0) * Quantity) AS TotalOrderValue
FROM
	OrderDetails
GROUP BY
	OrderID;

After executing the above query, I got the expected output as shown in the screenshot below.

From the output, the total value for each order is computed. For example, the total value for the order ID equal to 2 is 40.00, and for 5, it is 25.00.

But here, you need to understand how missing values are handled when computing the total value for each order.

Understand the query part, SUM(COALESCE(ItemPrice, 0) * Quantity); this part computes the total value for each order by multiplying each item’s price (ItemPrice) by its quantity (Quantity), then summing the products for all items in each order.

If ItemPrice is null, COALESCE() replaces it with 0 to ensure the computation can process without errors.

Here, in the COALESCE(ItemPrice, 0), this function returns the first non-value. It checks if ItemPrice contains a null value and then uses 0 as a fallback.

This is important for ensuring that multiplying by Qunatity doesn’t result in a null value, which would happen if ItemPrice were null. Using 0, the computation contributes nothing to the sum for items without a specified price rather than excluding them or causing an error.

This is how to use the SQL Server COALESCE function to handle the missing value in a table column.

Conclusion

In this SQL Server tutorial, you learned how to return the first non-null value from the list of values using the SQL Server COALESCE function.

Additionally, you have used the COALESCE() function on the table to handle null values in the column where you replaced the null with 0 value to ensure smooth operation without any error in the result.

You may like to read:

• SQL MIN MAX
• ISNULL Function in SQL Server
• CASE Statement in SQL Server

SQL COUNT DISTINCT

What Is SQL COUNT DISTINCT and How Does It Work?

In Structured Query Language (SQL), the standard COUNT() function is an aggregate function that returns the total number of rows matching a specific criterion. By default, COUNT(column_name) counts every non-null entry, including duplicates.

When you inject the DISTINCT keyword inside the aggregate function—COUNT(DISTINCT column_name)—you instruct the database query engine to eliminate duplicate values from the evaluation set before computing the final tally.

Key Conceptual Differences

Basic Syntax and Core Mechanics

The foundational ANSI SQL syntax for COUNT DISTINCT follows this standard pattern:

SQL

SELECT 
    COUNT(DISTINCT column_name) AS unique_count
FROM 
    table_name
WHERE 
    filter_conditions;

Example: Retail Customer Orders

To illustrate how the database engine processes this operation, consider a mock transactional table named CustomerOrders representing purchases across various US fulfillment hubs:

Total Orders vs. Unique Customers

If you want to compare the total volume of transactions against the actual unique customer count:

SQL

SELECT 
    COUNT(OrderID) AS total_transactions,
    COUNT(CustomerName) AS non_null_customers,
    COUNT(DISTINCT CustomerName) AS unique_customers
FROM 
    CustomerOrders;

Expected Output as shown below:

After executing the above query, I got the expected output as shown in the screenshot below.

Breakdown of Output:

• total_transactions (7): Evaluates every individual order record.
• unique_customers (4): The query engine groups Michael Carter, Emily Davis, Sarah Jenkins, and Robert Taylor, discarding the repeat transactions from Michael and Emily.

How SQL COUNT DISTINCT Handles NULL Values

A critical point of failure in data reporting involves misunderstandings around how COUNT(DISTINCT column_name) handles NULL records.

Crucial Rule: In standard ANSI SQL, COUNT(DISTINCT column_name) strictly ignores NULL values. It does not count NULL as a unique distinct entity.

Let us inspect a sample ClientAccounts table where some account managers are not yet assigned:

If you execute:

SQL

SELECT 
    COUNT(DISTINCT AccountManager) AS distinct_managers
FROM 
    ClientAccounts;

The result is 2 (David Miller and Jennifer White). The two records with NULL are filtered out during aggregate calculation.

What If You Need to Count NULL as a Distinct Value?

If your business requirements state that an unassigned status (NULL) represents an explicit, distinct classification state, you must transform NULL values into a placeholder using standard functions like COALESCE() or NVL():

SQL

SELECT 
    COUNT(DISTINCT COALESCE(AccountManager, 'Unassigned')) AS distinct_manager_states
FROM 
    ClientAccounts;

This returns 3 (David Miller, Jennifer White, and 'Unassigned').

Combining COUNT DISTINCT with GROUP BY

In analytical workflows, you rarely count distinct values across an entire table in isolation. More frequently, you slice distinct metrics across categorical dimensions like geographic regions, departments, or calendar quarters.

Scenario: Unique Customers per Region

Using our customer model, suppose we want to determine the number of distinct shoppers per state alongside total spend:

SQL

SELECT 
    State,
    COUNT(OrderID) AS total_orders,
    COUNT(DISTINCT CustomerName) AS unique_shoppers,
    SUM(OrderAmount) AS gross_revenue
FROM 
    CustomerOrders
GROUP BY 
    State
ORDER BY 
    gross_revenue DESC;

How the Database Processes This Query:

1. Partitioning: The engine splits the data into intermediate partitions based on State.
2. Aggregation: Within each partition, it sorts/hashes the CustomerName values to eliminate duplicates.
3. Computation: It computes the counts and sums independently for each state grouping.

Handling Multiple Columns in COUNT DISTINCT

For example, finding the number of distinct customer-to-state pairs or distinct department-and-role configurations.

The ANSI SQL Multi-Column Syntax

Standard ANSI SQL allows multiple column arguments within COUNT(DISTINCT ...):

SQL

-- Valid in PostgreSQL, MySQL, Oracle, and Snowflake
SELECT 
    COUNT(DISTINCT CustomerName, State) AS unique_customer_state_pairs
FROM 
    CustomerOrders;

The Microsoft SQL Server (T-SQL) Limitation and Solution

If you run the query above in Microsoft SQL Server (T-SQL), the database engine will throw an error:

Msg 102: Incorrect syntax near ','.

SQL Server does not natively support multiple column arguments inside a single COUNT(DISTINCT ...) call. To resolve this limitation in T-SQL, you can use two reliable patterns:

Method A: String Concatenation with Delimiters (Fast & Simple)

SQL

SELECT 
    COUNT(DISTINCT CustomerName + '|#|' + State) AS unique_customer_state_pairs
FROM 
    CustomerOrders;

Note: Always use an unambiguous delimiter (like |#|) to prevent false collision matches between columns (e.g., 'John' + 'Smith' vs. 'JohnS' + 'mith').

Method B: Subquery or Common Table Expression (CTE) (Clean & Robust)

SQL

WITH DistinctPairs AS (
    SELECT DISTINCT 
        CustomerName, 
        State
    FROM 
        CustomerOrders
)
SELECT 
    COUNT(*) AS unique_customer_state_pairs
FROM 
    DistinctPairs;

Method B is my preferred enterprise design pattern: it avoids string concatenation overhead, prevents character encoding issues, and produces self-documenting code that is easy for peer developers to maintain.

Conditional Aggregation with COUNT DISTINCT

In advanced reporting, you often need to calculate distinct values based on conditional filters without filtering out the entire query dataset via a restrictive WHERE clause.

Using CASE Statements Inside COUNT DISTINCT

Because COUNT(DISTINCT ...) ignores NULL, you can pair it with a CASE statement. When the condition evaluates to false, omit the ELSE branch (which defaults to NULL):

SQL

SELECT 
    COUNT(DISTINCT CustomerName) AS total_unique_customers,
    
    -- Count distinct customers who placed large orders
    COUNT(DISTINCT CASE 
        WHEN OrderAmount >= 100.00 THEN CustomerName 
    END) AS high_value_customers,
    
    -- Count distinct customers from specific southern states
    COUNT(DISTINCT CASE 
        WHEN State IN ('Texas', 'Florida') THEN CustomerName 
    END) AS southern_customers
FROM 
    CustomerOrders;

Why This Works:

• When OrderAmount < 100.00, the CASE statement returns NULL.
• COUNT(DISTINCT ...) automatically discards all NULL outputs, leaving only the distinct customers matching the targeted business rule.

Performance Pitfalls and Query Optimization

While COUNT(DISTINCT ...) is simple to write, it is among the most resource-intensive aggregate operations in relational database engines.

Why Is COUNT DISTINCT Expensive?

1. Sort and Hash Operations: Unlike standard COUNT(*), which simply increments a memory counter as it scans rows, COUNT(DISTINCT ...) requires the engine to keep track of every unique value encountered. It must build an in-memory hash table or sort the data stream to identify duplicates.
2. Memory Spills (TempDB / Disk): When processing tables with hundreds of millions of rows and high cardinality, the hash table often exceeds the available working memory buffer (work_mem in PostgreSQL or query workspace memory in SQL Server), forcing expensive I/O spills to disk.
3. Multiple COUNT DISTINCT Bottlenecks: Placing multiple distinct aggregations in a single SELECT list forces the optimizer to perform multiple distinct sorting passes over the same dataset:

SQL

-- Performance Warning: Requires multiple sorting/hashing passes
SELECT 
    Department,
    COUNT(DISTINCT EmployeeID) AS unique_staff,
    COUNT(DISTINCT ProjectCode) AS unique_projects,
    COUNT(DISTINCT VendorID) AS unique_vendors
FROM 
    EnterpriseOperations
GROUP BY 
    Department;

Proactive Optimization Strategies

• 1. Leverage Composite Indexes:Create covering indexes on the columns involved in the GROUP BY and COUNT(DISTINCT ...) clauses. A B-Tree index keeps data pre-sorted, allowing the engine to perform index stream aggregate scans without runtime sorting.
• 2. Pre-Aggregate with CTEs or Derived Tables:Deduplicate high-volume datasets early in the execution plan prior to performing complex multi-table joins.
• 3. Use Approximate Counting on Big Data Platforms:When querying massive analytical warehouses (Snowflake, Google BigQuery, AWS Redshift, or Databricks), exact distinct counts can be cost-prohibitive. In scenarios where a ~1% error margin is acceptable (e.g., high-level trend reporting), switch to HyperLogLog approximation functions:
  • Snowflake / BigQuery: APPROX_COUNT_DISTINCT(column_name)
  • SQL Server: APPROX_COUNT_DISTINCT(column_name)
  • AWS Redshift: COUNT(DISTINCT ...) with HyperLogLog extensions

Frequently Asked Questions (FAQ)

What is the difference between SELECT DISTINCT COUNT(col) and SELECT COUNT(DISTINCT col)?

• SELECT COUNT(DISTINCT col) counts the number of unique items in that column and returns a single integer.
• SELECT DISTINCT COUNT(col) first counts all non-null rows in the table (returning a single aggregate total) and then applies DISTINCT to that single number, which has no practical effect. Always use COUNT(DISTINCT col).

Can I use COUNT(DISTINCT) with window functions (OVER() clause)?

In standard SQL, most engines (including SQL Server and PostgreSQL) do not support COUNT(DISTINCT col) OVER (PARTITION BY ...). If you need distinct windowed aggregates, use DENSE_RANK() or compute the distinct values inside a Common Table Expression before applying window calculations.

Does COUNT(DISTINCT) count empty strings ('')?

Yes. An empty string ('') is a valid non-null string value in ANSI SQL. If your dataset has three blank strings and two distinct names, COUNT(DISTINCT col) will evaluate the blank string as one distinct entity.

Conclusion and Key Takeaways

The SQL COUNT DISTINCT function is an indispensable component of any data professional’s SQL toolkit. Understanding its internal logic ensures your analytical reporting remains accurate and computationally efficient.

Summary:

1. Always account for NULL values: Remember that COUNT(DISTINCT ...) naturally ignores NULL unless wrapped in a fallback function like COALESCE.
2. Mind your dialect constraints: Use CTEs or delimited concatenations when operating within SQL Server environments that restrict multi-column distinct arguments.
3. Optimize deliberately: Monitor performance bottlenecks on large tables by indexing properly, deduplicating early in subqueries, and using approximate algorithms (APPROX_COUNT_DISTINCT) on enterprise big data platforms.

You may also like the following articles:

• SQL Scalar Functions
• SQL PARTITION BY
• SQL UNPIVOT