Sending SMS messages one by one is straightforward, but scaling to hundreds or thousands requires a robust architecture. This guide details how to build a production-ready bulk SMS broadcasting system using Next.js for the application layer and Twilio Programmable Messaging for reliable delivery at scale.
We'll build a Next.js application with an API endpoint capable of accepting a list of phone numbers and a message body, then efficiently sending that message to all recipients via Twilio's Messaging Services. This approach solves the challenges of rate limiting, sender number management, and compliance handling inherent in bulk messaging.
Project Overview and Goals
What We're Building:
- A Next.js application (using the App Router).
- An API endpoint (
/api/broadcast
) to initiate bulk SMS sends. - Integration with Twilio Programmable Messaging, specifically using Messaging Services for scalability and compliance.
- A basic database schema (using PostgreSQL and Prisma) to define models for logging (
Broadcast
,MessageStatus
). Note: This guide focuses on passing the recipient list directly via the API request rather than storing and retrieving it from the database for the broadcast operation itself. - Webhook endpoint (
/api/webhooks/twilio
) to receive message status updates from Twilio. - Basic security using an API key.
- Deployment guidance using Vercel.
Problem Solved: Provides a scalable and manageable way to send the same SMS message to a large list of recipients without manually iterating API calls in a way that hits rate limits or requires complex sender logic. Leverages Twilio's infrastructure for queuing, delivery optimization, and compliance features (like opt-out handling).
Technologies:
- Next.js: Full-stack React framework for building the UI (minimal in this guide) and API layer.
- Twilio Programmable Messaging: For sending SMS messages via its robust API.
- Twilio Messaging Services: Essential for bulk messaging – handles sender pools, geo-matching, sticky sender, opt-out management, and queuing.
- PostgreSQL: A powerful open-source relational database (adaptable to others supported by Prisma).
- Prisma: Next-generation ORM for database access and migrations.
- TypeScript: For type safety and improved developer experience.
- Vercel: Platform for deploying Next.js applications.
System Architecture:
[User/Client] ----(HTTP POST Request)----> [Next.js API Route (/api/broadcast)]
| |
| | 1. Validates Request
| | 2. Iterates recipient list
| | 3. For each recipient:
| | - Calls Twilio API (messages.create)
| | using Messaging Service SID
|<----(HTTP Response - Success/Failure)-------|
|
| [Twilio Platform] <----(API Call)---- |
| |
| | Queues & Sends SMS via Messaging Service
| | (Handles sender selection_ rate limits_ etc.)
| |
| v
| [Carrier Networks] ----> [Recipient Phone]
|
| |
| | Status Update (sent, delivered, failed, etc.)
|<----(Webhook POST Request)----------| ----(Configured Callback URL)----> [Next.js API Route (/api/webhooks/twilio)]
|
| Processes status update
| (e.g., logs, updates DB)
Prerequisites:
- Node.js (v18 or later recommended) and npm/yarn/pnpm.
- A free or paid Twilio account.
- A Twilio phone number with SMS capabilities (purchased through the Twilio console).
- Access to a PostgreSQL database (local or cloud-hosted).
- Basic familiarity with TypeScript, Next.js, and REST APIs.
- Git installed.
- Vercel account (optional, for deployment).
Expected Outcome: A deployed Next.js application with a secure API endpoint that can reliably initiate bulk SMS broadcasts via Twilio and receive delivery status updates.
1. Setting up the Project
Let's initialize our Next.js project and install necessary dependencies.
1.1. Create Next.js Project:
Open your terminal and run the following command, choosing options suitable for this guide (TypeScript, Tailwind CSS, App Router):
npx create-next-app@latest twilio-bulk-sms
# When prompted:
# > Would you like to use TypeScript? Yes
# > Would you like to use ESLint? Yes
# > Would you like to use Tailwind CSS? Yes
# > Would you like to use `src/` directory? No (or Yes, adjust paths accordingly)
# > Would you like to use App Router? (recommended) Yes
# > Would you like to customize the default import alias? No
Navigate into the project directory:
cd twilio-bulk-sms
1.2. Install Dependencies:
We need the Twilio Node.js helper library, Prisma for database interaction, and Zod for validation. create-next-app
with TypeScript selected will typically include necessary development dependencies like @types/node
and typescript
.
npm install twilio prisma @prisma/client zod
1.3. Setup Prisma:
Initialize Prisma with PostgreSQL as the provider:
npx prisma init --datasource-provider postgresql
This creates:
prisma/schema.prisma
: Your database schema definition file..env
: A file for environment variables (Prisma automatically addsDATABASE_URL
).
1.4. Configure Environment Variables:
Open the .env
file created by Prisma. It will initially contain the DATABASE_URL
. We need to add our Twilio credentials and a secret key for our API.
Important: Never commit your .env
file to Git. Add .env
to your .gitignore
file if it's not already there.
# .env
# Database Connection (replace placeholder with your actual connection string)
# Example format: postgresql://USER:PASSWORD@HOST:PORT/DATABASE?schema=public
DATABASE_URL="postgresql://USER:PASSWORD@HOST:PORT/DATABASE?schema=public"
# Twilio Credentials (Obtain from Twilio Console: Account > API keys & tokens)
TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
TWILIO_AUTH_TOKEN="your_auth_token_xxxxxxxxxxxxxx"
# Twilio Messaging Service SID (Create in Twilio Console: Messaging > Services)
TWILIO_MESSAGING_SERVICE_SID="MGxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# Secret key for securing the broadcast API endpoint
API_SECRET_KEY="generate_a_strong_random_secret_key"
# Public URL of your application (used for webhook callback)
# Example: http://localhost:3000 for local dev, https://your-app.vercel.app for production
NEXT_PUBLIC_APP_URL="http://localhost:3000"
DATABASE_URL
: Important: Replace the placeholder value with your actual PostgreSQL connection string. Ensure the database exists.TWILIO_ACCOUNT_SID
/TWILIO_AUTH_TOKEN
: Find these in your Twilio Console under Account Info (Dashboard) or Account > API keys & tokens.TWILIO_MESSAGING_SERVICE_SID
: We will create this in the Twilio Integration section (Step 4). You can leave it blank for now or add a placeholder likeMGxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.API_SECRET_KEY
: Generate a strong, unique random string (e.g., using a password manager or online generator). This will be used to authenticate requests to our broadcast API.NEXT_PUBLIC_APP_URL
: Set this to your application's base URL. It's needed for constructing the webhook callback URL. Usehttp://localhost:3000
for local development and your deployed URL (e.g.,https://your-app.vercel.app
) for production.
1.5. Define Database Schema:
Open prisma/schema.prisma
and define models for logging broadcasts and message statuses.
// prisma/schema.prisma
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
}
// Optional: Model to store recipient details if managing lists within the app
// model Recipient {
// id String @id @default(cuid())
// phoneNumber String @unique // Store in E.164 format e.g., +14155552671
// name String?
// optedIn Boolean @default(true)
// createdAt DateTime @default(now())
// updatedAt DateTime @updatedAt
// }
model Broadcast {
id String @id @default(cuid())
messageBody String
recipientCount Int
status String @default("initiated") // e.g., initiated, processing, submitted, partial_failure, failed, completed (derived from statuses)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
MessageStatus MessageStatus[] // Relation to individual message statuses
}
model MessageStatus {
id String @id @default(cuid())
messageSid String @unique // Twilio's Message SID
broadcastId String? // Link back to the broadcast job
status String // e.g., queued, sending, sent, delivered, undelivered, failed
to String // Recipient phone number (E.164)
errorCode Int? // Twilio error code if failed/undelivered
errorMessage String?
receivedAt DateTime @default(now()) // When the webhook was received/processed
updatedAt DateTime @updatedAt
// Relation to Broadcast model
broadcast Broadcast? @relation(fields: [broadcastId], references: [id])
// Index for querying by status or broadcast ID
@@index([status])
@@index([broadcastId])
}
1.6. Apply Database Migrations:
Run the following command to create the necessary SQL migration files and apply them to your database:
npx prisma migrate dev --name init
This will:
- Create an SQL migration file in
prisma/migrations/
. - Apply the migration to your database, creating the
Broadcast
andMessageStatus
tables. - Generate the Prisma Client based on your schema (
@prisma/client
).
1.7. Initialize Twilio Client:
Create a utility file to initialize the Twilio client instance.
// lib/twilioClient.ts
import twilio from 'twilio';
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
if (!accountSid || !authToken) {
throw new Error('Twilio credentials (TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN) are not set in environment variables.');
}
const client = twilio(accountSid, authToken);
export default client;
1.8. Initialize Prisma Client:
Create a utility file for the Prisma client instance.
// lib/prisma.ts
import { PrismaClient } from '@prisma/client';
declare global {
// allow global `var` declarations
// eslint-disable-next-line no-var
var prisma: PrismaClient | undefined;
}
export const prisma =
global.prisma ||
new PrismaClient({
// Optional: Enable logging for development
// log: ['query', 'info', 'warn', 'error'],
});
if (process.env.NODE_ENV !== 'production') global.prisma = prisma;
Project setup is complete. We have Next.js running, dependencies installed, environment variables configured, the database schema defined and migrated, and utility files for Twilio/Prisma clients.
2. Implementing Core Functionality (Simple Test Send)
Before tackling bulk sending, let's create a simple API route to verify our Twilio connection by sending a single message using a specific from
number. This ensures credentials and basic API interaction are working.
2.1. Create Simple Send API Route:
Create the file app/api/send-simple/route.ts
:
// app/api/send-simple/route.ts
import { NextResponse } from 'next/server';
import twilioClient from '@/lib/twilioClient';
// **Warning:** The following `TWILIO_PHONE_NUMBER` constant is a *placeholder*.
// You **must** replace `'+15017122661'` with your actual, purchased Twilio phone number
// capable of sending SMS for this test to work.
const TWILIO_PHONE_NUMBER = '+15017122661'; // REPLACE THIS with your actual Twilio number
export async function POST(request: Request) {
console.log('Received request for /api/send-simple');
if (!process.env.TWILIO_ACCOUNT_SID || !process.env.TWILIO_AUTH_TOKEN) {
console.error('Server configuration error: Twilio credentials not configured');
return NextResponse.json({ error: 'Twilio credentials not configured' }, { status: 500 });
}
if (!TWILIO_PHONE_NUMBER || TWILIO_PHONE_NUMBER === '+15017122661') {
console.error('Error: TWILIO_PHONE_NUMBER constant is not set or is still the placeholder. Please edit app/api/send-simple/route.ts.');
return NextResponse.json({ error: 'Server configuration error: Twilio phone number not set.' }, { status: 500 });
}
try {
const body = await request.json();
const { to, message } = body;
// Basic validation
if (!to || !message) {
return NextResponse.json({ error: 'Missing `to` or `message` in request body' }, { status: 400 });
}
// Validate 'to' number format (basic check, E.164 required)
if (!/^\+[1-9]\d{1,14}$/.test(to)) {
return NextResponse.json({ error: 'Invalid `to` phone number format. Use E.164 (e.g., +14155552671)' }, { status: 400 });
}
console.log(`Attempting to send message to: ${to} from ${TWILIO_PHONE_NUMBER}`);
const twilioResponse = await twilioClient.messages.create({
body: message,
from: TWILIO_PHONE_NUMBER, // Using a specific purchased number for this test
to: to, // Recipient number from request body
});
console.log('Twilio message SID:', twilioResponse.sid);
return NextResponse.json({ success: true, messageSid: twilioResponse.sid });
} catch (error: any) {
console.error('Error sending message via Twilio:', error);
// Provide more specific error feedback if possible
const errorMessage = error.message || 'Failed to send message';
const statusCode = error.status || 500; // Use Twilio's status code if available
return NextResponse.json({ error: `Failed to send message: ${errorMessage}`, code: error.code }, { status: statusCode });
}
}
2.2. Testing the Simple Send:
You can test this using curl
or a tool like Postman. Replace YOUR_VERCEL_URL
with http://localhost:3000
during local development, and <YOUR_PHONE_NUMBER_E164>
with your actual mobile number in E.164 format (e.g., +14155551234
).
curl -X POST YOUR_VERCEL_URL/api/send-simple \
-H ""Content-Type: application/json"" \
-d '{
""to"": ""<YOUR_PHONE_NUMBER_E164>"",
""message"": ""Hello from Next.js and Twilio simple send!""
}'
Expected Response (Success):
{
""success"": true,
""messageSid"": ""SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxx""
}
You should also receive the SMS on your phone shortly. If you encounter errors, check:
- Your
.env
file has the correct Twilio Account SID and Auth Token. - The
TWILIO_PHONE_NUMBER
constant inapp/api/send-simple/route.ts
has been replaced with a number you own in Twilio. - The
to
number in your test command is valid and in E.164 format. - Your Next.js development server (
npm run dev
) is running. - Check the terminal logs for specific error messages from Twilio (including error codes).
3. Building the Bulk Broadcast API Layer
Now, let's build the core API endpoint (/api/broadcast
) that will handle sending messages to multiple recipients using a Messaging Service.
3.1. Define Request Validation Schema:
Using Zod, we define the expected shape of the request body.
// lib/validators.ts
import { z } from 'zod';
// Basic E.164 format validation (adjust regex as needed for stricter validation)
const phoneSchema = z.string().regex(/^\+[1-9]\d{1,14}$/, {
message: "Phone number must be in E.164 format (e.g., +14155552671)",
});
export const broadcastSchema = z.object({
recipients: z.array(phoneSchema).min(1, "At least one recipient is required"),
message: z.string().min(1, "Message body cannot be empty").max(1600, "Message body is too long"), // Twilio segment limit
});
export type BroadcastPayload = z.infer<typeof broadcastSchema>;
3.2. Create Broadcast API Route:
Create the file app/api/broadcast/route.ts
. This route will:
- Check for an
Authorization
header containing ourAPI_SECRET_KEY
. - Validate the request body using the Zod schema.
- Retrieve the
TWILIO_MESSAGING_SERVICE_SID
andNEXT_PUBLIC_APP_URL
from environment variables. - Log the broadcast attempt to the database (
Broadcast
table). - Iterate through the recipient list and send a message to each using the Messaging Service SID and configure the status callback URL.
- Handle potential errors during the process.
// app/api/broadcast/route.ts
import { NextResponse } from 'next/server';
import twilioClient from '@/lib/twilioClient';
import { prisma } from '@/lib/prisma';
import { broadcastSchema, BroadcastPayload } from '@/lib/validators';
import { ZodError } from 'zod';
const API_SECRET_KEY = process.env.API_SECRET_KEY;
const MESSAGING_SERVICE_SID = process.env.TWILIO_MESSAGING_SERVICE_SID;
const APP_URL = process.env.NEXT_PUBLIC_APP_URL;
export async function POST(request: Request) {
console.log('Received request for /api/broadcast');
// 1. Authentication
const authHeader = request.headers.get('Authorization');
if (!API_SECRET_KEY) {
console.error('CRITICAL: API_SECRET_KEY is not set in environment variables.');
return NextResponse.json({ error: 'Server configuration error: Auth mechanism not configured.' }, { status: 500 });
}
// TODO: Use timing-safe comparison for production
if (!authHeader || authHeader !== `Bearer ${API_SECRET_KEY}`) {
console.warn('Unauthorized broadcast attempt');
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}
// Check required Twilio configurations
if (!MESSAGING_SERVICE_SID) {
console.error('CRITICAL: TWILIO_MESSAGING_SERVICE_SID is not set in environment variables.');
return NextResponse.json({ error: 'Server configuration error: Messaging Service SID missing.' }, { status: 500 });
}
if (!process.env.TWILIO_ACCOUNT_SID || !process.env.TWILIO_AUTH_TOKEN) {
console.error('CRITICAL: Twilio Account SID or Auth Token missing.');
return NextResponse.json({ error: 'Server configuration error: Twilio credentials missing.' }, { status: 500 });
}
if (!APP_URL) {
console.error('CRITICAL: NEXT_PUBLIC_APP_URL is not set. Cannot set status callback URL.');
return NextResponse.json({ error: 'Server configuration error: Application URL missing.' }, { status: 500 });
}
let payload: BroadcastPayload;
try {
// 2. Validation
const rawBody = await request.json();
payload = broadcastSchema.parse(rawBody);
console.log(`Validated broadcast request for ${payload.recipients.length} recipients.`);
} catch (error) {
if (error instanceof ZodError) {
console.error('Validation error:', error.errors);
return NextResponse.json({ error: 'Invalid request body', details: error.format() }, { status: 400 });
}
console.error('Error parsing request body:', error);
return NextResponse.json({ error: 'Bad Request' }, { status: 400 });
}
const { recipients, message } = payload;
let successfulSubmissions = 0;
let failedSubmissions = 0;
const sendPromises: Promise<any>[] = [];
const startTime = Date.now();
const statusCallbackUrl = `${APP_URL}/api/webhooks/twilio`;
// 3. Log Broadcast Attempt
let broadcastRecord;
try {
broadcastRecord = await prisma.broadcast.create({
data: {
messageBody: message,
recipientCount: recipients.length,
status: 'processing', // Initial status
},
});
console.log(`Created broadcast record: ${broadcastRecord.id}`);
} catch (dbError) {
console.error("Failed to create broadcast record in DB:", dbError);
// Decide if you want to proceed without logging or return an error
// For this guide, we'll log the error but proceed with sending
// return NextResponse.json({ error: 'Database error logging broadcast' }, { status: 500 });
}
// 4. Iterate and Send using Messaging Service
console.log(`Initiating send to ${recipients.length} recipients via Messaging Service: ${MESSAGING_SERVICE_SID}`);
recipients.forEach((recipient) => {
const promise = twilioClient.messages
.create({
body: message,
messagingServiceSid: MESSAGING_SERVICE_SID, // *** Use Messaging Service SID ***
to: recipient,
// Define the Status Callback URL to receive status updates
statusCallback: statusCallbackUrl,
})
.then((msg) => {
successfulSubmissions++;
console.log(`Successfully submitted message to ${recipient}, SID: ${msg.sid}`);
// Optionally log individual message submission status here or rely on webhooks for final status
// Associate message SID with broadcast ID immediately if possible (though webhook is more reliable for final status)
if (broadcastRecord) {
prisma.messageStatus.create({
data: {
messageSid: msg.sid,
broadcastId: broadcastRecord.id,
status: msg.status, // Initial status from Twilio (e.g., 'queued', 'accepted')
to: recipient,
}
}).catch(dbError => console.error(`Failed to log initial status for ${msg.sid}:`, dbError));
}
})
.catch((error) => {
failedSubmissions++;
console.error(`Failed to submit message to ${recipient}: ${error.message} (Code: ${error.code})`);
// Optionally log individual message failure status here
// Common submission errors: 21211 (Invalid 'To'), 21610 (Opt-out), 21612 (Msg Service unavailable)
if (broadcastRecord) {
prisma.messageStatus.create({
data: {
messageSid: `failed_submission_${recipient}_${Date.now()}`, // Placeholder SID for failed submissions
broadcastId: broadcastRecord.id,
status: 'failed_submission',
to: recipient,
errorCode: error.code,
errorMessage: error.message,
}
}).catch(dbError => console.error(`Failed to log failed submission for ${recipient}:`, dbError));
}
});
sendPromises.push(promise);
});
// 5. Wait for all submission requests to Twilio to complete (or fail)
// Note: This only confirms Twilio *accepted* the request to send, not delivery.
await Promise.allSettled(sendPromises);
const duration = Date.now() - startTime;
console.log(`Broadcast submission processing finished in ${duration}ms. Success: ${successfulSubmissions}, Failed: ${failedSubmissions}`);
// 6. Update Broadcast Record Status (Optional)
if (broadcastRecord) {
try {
const finalStatus = failedSubmissions > 0
? (successfulSubmissions > 0 ? 'partial_failure' : 'failed')
: 'submitted'; // Indicate submission complete, final status via webhooks
await prisma.broadcast.update({
where: { id: broadcastRecord.id },
data: {
status: finalStatus,
},
});
} catch (dbError) {
console.error("Failed to update broadcast record status in DB:", dbError);
}
}
// 7. Return Response
return NextResponse.json({
message: `Broadcast initiated. Submitted to Twilio: ${successfulSubmissions}, Failed submissions: ${failedSubmissions}`,
broadcastId: broadcastRecord?.id ?? null, // Include ID if logged
});
}
Explanation:
- Authentication: Checks for
Authorization: Bearer <YOUR_API_SECRET_KEY>
. Added check forAPI_SECRET_KEY
existence. - Validation: Uses Zod to ensure
recipients
is an array of valid E.164 strings andmessage
is present. - Messaging Service SID: Crucially, it uses
messagingServiceSid
instead offrom
. This tells Twilio to use the pool of numbers and rules defined in that service. - Status Callback: Added the
statusCallback
parameter pointing to our future webhook endpoint, constructed usingNEXT_PUBLIC_APP_URL
. - Iteration: It loops through each recipient and fires off an asynchronous request to Twilio via
twilioClient.messages.create
. Promise.allSettled
: Waits for all these initial API calls to Twilio to either succeed (Twilio accepted the request) or fail (e.g., invalid SID, auth error, invalid 'To' number for submission). It does not wait for the SMS messages to be delivered.- Logging: Creates a record in the
Broadcast
table before sending. It now also attempts to create an initialMessageStatus
record upon successful submission or logs a failure record immediately on submission error, linking them to thebroadcastId
. Updates theBroadcast
status after attempting all submissions. - Error Handling: Catches validation errors, Twilio API errors during the loop, and database errors. Logs error codes from Twilio.
4. Integrating with Twilio Messaging Services
Using a Messaging Service is key for bulk sending. It manages sender phone numbers, handles opt-outs, ensures compliance, and intelligently routes messages.
4.1. Create a Messaging Service in Twilio:
- Go to the Twilio Console.
- Navigate to Messaging > Services.
- Click ""Create Messaging Service"".
- Give it a recognizable name (e.g., ""Next.js Broadcast Service"").
- Select the use case – ""Notifications"", ""Marketing"", or ""Customer Care"" often fit bulk scenarios. Choose the one that best describes your intended use. Configure compliance info if prompted (important for A2P 10DLC registration if using US 10-digit numbers).
- Click ""Create"".
4.2. Add Sender Numbers:
- Once the service is created, you'll be on its configuration page. Click on ""Sender Pool"" in the left menu.
- Click ""Add Senders"".
- Select ""Phone Number"" as the Sender Type. Click ""Continue"".
- Choose the Twilio phone number(s) you want to use for sending bulk messages from this service. You must add at least one number. Using multiple numbers allows Twilio to distribute the load (sender rotation).
- Click ""Add Phone Numbers"".
4.3. Configure Compliance (A2P 10DLC - US Specific):
- If sending to the US using standard 10-digit long codes (10DLC), you must register for A2P 10DLC.
- Within the Messaging Service settings, go to the ""Compliance"" section (or look for A2P 10DLC settings) and link your approved A2P Brand and Campaign registration. This involves providing business details and use case information to Twilio for carrier approval.
- Failure to register will result in message filtering and non-delivery to US numbers. Consult Twilio's documentation on A2P 10DLC registration for detailed steps. Toll-Free numbers have different verification processes, and Short Codes require separate applications.
4.4. Configure Opt-Out Management:
- In the Messaging Service settings, go to ""Opt-Out Management"".
- Ensure ""STOP/HELP/START Handling"" (or similar wording like ""Twilio Advanced Opt-Out"") is enabled. Twilio will automatically handle standard opt-out (STOP, UNSUBSCRIBE) and opt-in (START) keywords and help requests (HELP). This is crucial for compliance (e.g., TCPA in the US).
- You can customize the confirmation messages if needed.
4.5. Obtain the Messaging Service SID:
- Go back to the main Messaging > Services list in the Twilio Console.
- Find the service you just created.
- Copy the ""SERVICE SID"" (it starts with
MG...
).
4.6. Update Environment Variable:
Paste the copied Service SID into your .env
file:
# .env
# ... other variables
TWILIO_MESSAGING_SERVICE_SID=""MGxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"" # Paste the SID here
# ...
Restart your Next.js development server (npm run dev
) for the new environment variable to be loaded.
Your application is now configured to send messages through the Twilio Messaging Service, leveraging its scaling and compliance features.
5. Implementing Error Handling, Logging, and Status Webhooks
Robust error handling and understanding message status are vital.
5.1. Enhanced Error Handling (In API Route):
The /api/broadcast
route already includes try...catch
blocks and logs Twilio error codes during submission. Key improvements:
- Specific Twilio Errors: Logging
error.code
anderror.message
from Twilio helps diagnose issues (e.g.,21211
- Invalid 'To' phone number,20003
- Authentication error,21610
- Attempt to send to unsubscribed recipient). - Database Errors: Wrap Prisma calls in
try...catch
to handle database connection issues or constraint violations, logging errors without necessarily halting the entire process if appropriate. - Clear Logging: Use
console.log
,console.warn
,console.error
appropriately. In production, consider structured logging libraries (Pino, Winston) that output JSON for easier parsing by log management systems (like Vercel Logs, Datadog, etc.).
5.2. Implementing Status Webhooks:
Twilio can send HTTP requests (webhooks) to your application whenever the status of a message changes (e.g., queued
, sent
, delivered
, undelivered
, failed
). This is the only reliable way to know the final delivery status.
5.2.1. Create Webhook Handler API Route:
Create the file app/api/webhooks/twilio/route.ts
:
// app/api/webhooks/twilio/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { prisma } from '@/lib/prisma';
import { twilio } from 'twilio'; // Import to use the validator
const TWILIO_AUTH_TOKEN = process.env.TWILIO_AUTH_TOKEN;
// Helper function to get the full URL, handling potential proxy headers
function getAbsoluteUrl(req: NextRequest): string {
const protocol = req.headers.get('x-forwarded-proto') || (process.env.NODE_ENV === 'production' ? 'https' : 'http');
const host = req.headers.get('host'); // Host includes port if non-standard
const pathname = req.nextUrl.pathname;
const search = req.nextUrl.search; // Include query params if any
if (!host) {
// Fallback or error handling if host is missing
console.error("Webhook handler couldn't determine host header.");
// Attempt using NEXT_PUBLIC_APP_URL as a fallback base, but this might be less accurate
const appUrl = process.env.NEXT_PUBLIC_APP_URL || '';
return `${appUrl}${pathname}${search}`;
}
return `${protocol}://${host}${pathname}${search}`;
}
export async function POST(request: NextRequest) {
console.log('Received Twilio webhook');
if (!TWILIO_AUTH_TOKEN) {
console.error("CRITICAL: TWILIO_AUTH_TOKEN is not set for webhook validation. Cannot validate request.");
// Return 200 OK to Twilio even on server config error to prevent retries.
// Log the error internally. This is the recommended approach.
return new Response(null, { status: 200 });
// Avoid returning 5xx, as Twilio retries might worsen the problem. Rely on internal monitoring.
}
// 1. Validate Twilio Request (Security)
// Get headers needed for validation. Header names are case-insensitive per HTTP spec,
// but NextRequest provides them lower-cased via request.headers.get().
const signature = request.headers.get('x-twilio-signature');
const url = getAbsoluteUrl(request); // Use helper to reconstruct the full URL
const rawBody = await request.text(); // Get raw body for validation
// Reconstruct body as key/value pairs (Twilio webhooks are form-urlencoded)
const params = new URLSearchParams(rawBody);
const bodyParams: Record<string_ string> = {};
params.forEach((value, key) => {
bodyParams[key] = value;
});
if (!signature) {
console.warn('Webhook received without x-twilio-signature header.');
// Still return 200 to prevent retries, but log the issue.
return new Response('Missing validation signature header', { status: 400 }); // Use 400 Bad Request
}
let isValid = false;
try {
isValid = twilio.validateRequest(
TWILIO_AUTH_TOKEN,
signature,
url,
bodyParams // Pass the parsed key/value object
);
} catch (validationError) {
console.error("Error during Twilio validation:", validationError);
// Treat validation errors as invalid requests
isValid = false;
}
if (!isValid) {
console.warn(`Invalid Twilio webhook signature for URL: ${url}`);
// Return 403 Forbidden if the signature is invalid
return new Response('Invalid signature', { status: 403 });
}
console.log('Twilio webhook signature validated successfully.');
// 2. Process Status Update
const messageSid = bodyParams.MessageSid;
const messageStatus = bodyParams.MessageStatus; // delivered, undelivered, failed, sent, queued etc.
const to = bodyParams.To;
const errorCode = bodyParams.ErrorCode ? parseInt(bodyParams.ErrorCode, 10) : undefined;
const errorMessage = bodyParams.ErrorMessage;
if (!messageSid || !messageStatus || !to) {
console.warn('Webhook missing required parameters (MessageSid, MessageStatus, To)');
// Return 200 OK but log the issue
return new Response('Missing required parameters', { status: 400 });
}
console.log(`Status Update: SID=${messageSid}, Status=${messageStatus}, To=${to}, ErrorCode=${errorCode || 'N/A'}`);
try {
// 3. Store or Update Status in Database
// Use upsert: update if messageSid exists, create if it doesn't
// This handles cases where the initial submission log failed or webhook arrives first
const updatedStatus = await prisma.messageStatus.upsert({
where: { messageSid: messageSid },
update: {
status: messageStatus,
errorCode: errorCode,
errorMessage: errorMessage,
// updatedAt is handled automatically by Prisma @updatedAt
},
create: {
messageSid: messageSid,
status: messageStatus,
to: to,
errorCode: errorCode,
errorMessage: errorMessage,
// Attempt to find associated broadcastId if not already linked
// This requires a more complex lookup or passing broadcastId via custom Twilio params
// broadcastId: findBroadcastIdForMessage(messageSid), // Placeholder for lookup logic
}
});
console.log(`Stored/Updated status for ${messageSid}. New status: ${updatedStatus.status}`);
// Optional: Update overall Broadcast status based on aggregated MessageStatus updates
// This logic can be complex (e.g., check if all messages are final, calculate success rate)
// if (updatedStatus.broadcastId) {
// await updateBroadcastStatus(updatedStatus.broadcastId);
// }
} catch (dbError) {
console.error(`Database error processing webhook for ${messageSid}:`, dbError);
// IMPORTANT: Return 200 OK to Twilio even if DB write fails to prevent retries.
// Ensure robust internal logging/monitoring alerts on these DB errors.
// Acknowledge receipt so Twilio doesn't retry endlessly on a DB issue.
return new Response(null, { status: 200 });
}
// 4. Acknowledge Receipt to Twilio
// Return an empty 200 OK response to signal successful receipt.
return new Response(null, { status: 200 });
}
// Placeholder functions for potential enhancements
// async function findBroadcastIdForMessage(messageSid: string): Promise<string | null> {
// // Implement logic to find the broadcast ID, maybe based on custom parameters
// // passed during message creation or other contextual information.
// return null;
// }
// async function updateBroadcastStatus(broadcastId: string): Promise<void> {
// // Implement logic to check all MessageStatus records for the broadcast
// // and update the Broadcast record's status (e.g., to 'completed', 'failed').
// console.log(`Triggering status update check for broadcast ${broadcastId}`);
// }
Explanation:
- Validation: Uses
twilio.validateRequest
with theTWILIO_AUTH_TOKEN
, thex-twilio-signature
header, the reconstructed request URL, and the parsed form-urlencoded body parameters. This is crucial security to ensure the request genuinely came from Twilio. Includes a helpergetAbsoluteUrl
to handle URL reconstruction, especially behind proxies. - Error Handling: Returns
403 Forbidden
on invalid signature. Returns400 Bad Request
for missing parameters but crucially returns200 OK
even if there's a server configuration issue (missing Auth Token) or a database error during processing. This prevents Twilio from endlessly retrying the webhook due to downstream issues. Log these internal errors thoroughly. - Data Extraction: Parses the form-urlencoded body (
application/x-www-form-urlencoded
) sent by Twilio to getMessageSid
,MessageStatus
,To
,ErrorCode
, etc. - Database Update: Uses
prisma.messageStatus.upsert
to either create a new status record (if the webhook arrives before or instead of the initial submission log) or update the existing record for the givenmessageSid
. - Acknowledgement: Returns an empty
200 OK
response to Twilio upon successful validation and processing (or recoverable error) to prevent retries.
5.2.2. Exposing the Webhook URL:
For Twilio to reach your webhook:
- Local Development: Use a tool like
ngrok
(ngrok http 3000
) to expose your locallocalhost:3000
to the internet.ngrok
will give you a public URL (e.g.,https://<random-string>.ngrok.io
). UpdateNEXT_PUBLIC_APP_URL
in your.env
temporarily to thisngrok
URL. - Production (Vercel): Your deployed Vercel URL (e.g.,
https://your-app-name.vercel.app
) is already public. EnsureNEXT_PUBLIC_APP_URL
is set correctly in your Vercel project's environment variables.
5.2.3. Configuring the Webhook in Twilio (Optional but Recommended):
While we set the statusCallback
URL per message, you can also set a default webhook URL at the Messaging Service level in the Twilio Console (Messaging > Services > [Your Service] > Integration). This acts as a fallback if the per-message callback isn't provided or fails. Ensure the URL points to /api/webhooks/twilio
.
6. Deployment to Vercel
Deploying this Next.js application to Vercel is straightforward.
- Push to Git: Ensure your project is committed to a Git repository (GitHub, GitLab, Bitbucket). Make sure
.env
is in your.gitignore
. - Import Project in Vercel: Log in to Vercel, click ""Add New..."" > ""Project"", and import your Git repository.
- Configure Project: Vercel usually detects Next.js automatically.
- Configure Environment Variables: Go to the project settings in Vercel (Settings > Environment Variables). Add the same environment variables as in your
.env
file:DATABASE_URL
(use your production database connection string)TWILIO_ACCOUNT_SID
TWILIO_AUTH_TOKEN
TWILIO_MESSAGING_SERVICE_SID
API_SECRET_KEY
(use the same strong secret)NEXT_PUBLIC_APP_URL
(set this to your production Vercel domain, e.g.,https://your-app-name.vercel.app
)
- Deploy: Trigger a deployment (usually happens automatically on push to the main branch).
- Test: Once deployed, use
curl
or Postman to test your/api/broadcast
endpoint using the production URL and yourAPI_SECRET_KEY
. Check Vercel logs for output and Twilio console/debugger for message status. Ensure webhook calls are reaching your Vercel deployment and being processed correctly.
Conclusion and Next Steps
You now have a robust foundation for a bulk SMS broadcasting system using Next.js and Twilio Messaging Services. This setup handles scaling, sender management, basic compliance, and status tracking.
Potential Improvements:
- UI: Build a simple frontend page to trigger broadcasts instead of using
curl
. - Recipient Management: Implement features to upload, store, and manage recipient lists within the database (
Recipient
model). - Advanced Status Tracking: Update the main
Broadcast
record status based on aggregatedMessageStatus
updates (e.g., calculate completion percentage, final status). - Rate Limiting: Implement rate limiting on the
/api/broadcast
endpoint itself (e.g., usingupstash/ratelimit
). - Idempotency: Ensure webhook processing is idempotent (processing the same webhook multiple times doesn't cause issues).
upsert
helps here. - Detailed Error Reporting: Integrate an error tracking service (Sentry, etc.).
- Background Jobs: For very large lists (> thousands), consider offloading the iteration and Twilio API calls to a background job queue (e.g., Vercel Cron Jobs, BullMQ, Quirrel) instead of processing synchronously within the API route's request lifecycle. This prevents API timeouts.
- Security: Implement more robust authentication/authorization if needed. Use timing-safe comparisons for secrets.