# Create a Dynamic Sitemap with Next.js

One of the best ways to drive traffic to your website is to have strong Search Engine Optimization (SEO). You can provide search engines with all the URLs for your website using a **Sitemap**. This allows for easier indexing and more efficient crawling by the search engines.

Maintaining a static sitemap can be tedious and more work if your website is always changing. The best solution is to dynamically create one.

Let's check out a couple of ways we can accomplish this.

## Create a sitemap using a script at build time

If all of your content and pages are local in your project, you can easily use a script at build time to create a `sitemap.xml`.

My blog uses [MDX](https://mdxjs.com/) files instead of a CMS, so I do not have to worry about my content changing after build time.

My script uses `globby` to traverse the file system and return all my routes. First thing we need to do is install it as a dev dependency.

```
npm i -D globby
```

Then we can create the script:

`scripts/generate-sitemap.js`

```js
const fs = require('fs');
const globby = require('globby');

const generateSitemap = async () => {
  // Fetch all routes based on patterns
  // Your folder structure might be different so change bellow to match your needs
  const pages = await globby([
    'pages/**/*.{ts,tsx,mdx}', // All routes inside /pages
    '_content/**/*.mdx', // All MDX files inside my /_content
    '!pages/**/[*.{ts,tsx}', // Ignore my dynamic route index Example /pages/blog/[slug].tsx
    '!pages/_*.{ts,tsx}', // Ignore next.js files
    '!pages/api', // Ignore API routes
    '!pages/admin.tsx' // Ignore pages not meant to be indexed
  ]);

  const urlSet = pages
    .map((page) => {
      // Remove none route related parts of filename.
      const path = page
        .replace('pages', '')
        .replace('_content', '')
        .replace(/(.tsx|.ts)/, '')
        .replace('.mdx', '');
      // Remove the word index from route
      const route = path === '/index' ? '' : path;
      // Build url portion of sitemap.xml
      return `<url><loc>https://codebycorey.com${route}</loc></url>`;
    })
    .join('');

  // Add urlSet to entire sitemap string
  const sitemap = `<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${urlSet}</urlset>`;

  // Create sitemap file
  fs.writeFileSync('public/sitemap.xml', sitemap);
};

module.exports = generateSitemap;
```

To run the script at build time, you can create a `next.config.js` file. This will modify Next.js build process.

```js
const generateSitemap = require('./scripts/generate-sitemap');
const generateRSS = require('./scripts/generate-rss');

module.exports = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      generateSitemap();
    }
    return config;
  }
};
```

Now when you build your website, you should see a freshly created `public/sitemap.xml`.

Lastly, I recommend adding `public/sitemap.xml` to your `.gitignore` since it is a generated file.

## Create a sitemap using a route

You cannot create a `sitemap` at build time When you are using a content management system (CMS). It might work when you first build your project, but if you push out new content after the build, your `sitemap` will be outdated.

What we could do is create an API route to fetch the data, and we rewrite the sitemap request to use the API route.

First create the API route:

`pages/api/sitemap.ts`

```ts
export default async (req, res) => {
  // Fetch data from a CMS.
  const resp = await fetch('MOCK_URL');
  const externalPosts = await resp.json();

  const routes = externalPosts.map((post) => `/blog/${posts.slug}`);
  const localRoutes = ['/index', '/blog'];

  const pages = routes.concat(localRoutes);

  const urlSet = pages
    .map((page) => {
      // Remove none route related parts of filename.
      const path = page
        .replace('pages', '')
        .replace('_content', '')
        .replace(/(.tsx|.ts)/, '')
        .replace('.mdx', '');
      // Remove the word index from route
      const route = path === '/index' ? '' : path;
      // Build url portion of sitemap.xml
      return `<url><loc>https://codebycorey.com${route}</loc></url>`;
    })
    .join('');

  // Add urlSet to entire sitemap string
  const sitemap = `<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${urlSet}</urlset>`;

  // set response content header to xml
  res.setHeader('Content-Type', 'text/xml');
  // write the sitemap
  res.write(sitemap);
  res.end();
};
```

Now we can create a route rewrite to make `/sitemap.xml` actually call `/api/sitemap`.

Create `next.config.js` and add:

```js
module.exports = {
  async rewrites() {
    return [
      {
        source: '/sitemap.xml',
        destination: '/api/sitemap'
      }
    ];
  }
};
```

Now when you navigate to `http://localhost:3000/sitemap.xml`, you should see your sitemap based on local files and your CMS.

## Bonus: Robots.txt

One additional thing you can add to your website to improve SEO is a `robots.txt` (AKA robots exclusion standard). This basically tells search engines which routes they are not allowed to index.

Create `public/robots.txt` and add

```
User-agent: *
Disallow:

Sitemap: https://your-url.com/sitemap.xml
```

This will tell search engines they are welcome to crawl your entire website.

If you would like to block any pages from being indexed, add it as disallow field.

```
User-agent: *
Disallow: /admin
Disallow: /secret-page

Sitemap: https://your-url.com/sitemap.xml
```

---

- Follow me on [Twitter](https://twitter.com/CodeByCorey) for random posts about tech and programming. I am also documenting my journey learning design.
- Nest.js [Docs](https://nextjs.org/docs/getting-started)